2024-04-22T08:12:56.972 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T08:12:56.972 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875 --name teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi --verbose --description rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} -- /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/orig.config.yaml 2024-04-22T08:12:57.015 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T08:12:57.108 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875 branch: reef description: rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667875' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: c3a345d31ea77c01771428ab80e1cf32c9b05875 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 seed: 7353 sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d sleep_before_teardown: 0 subset: 3333/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c3a345d31ea77c01771428ab80e1cf32c9b05875 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPXa6y+CGC715PpLZE+5qeK4KRv0t7J8H9pbJCwdRLylAubUnXtl8lrHFVEQjGHTCUNDdJIqkf3B6PV4qkgJsc= tasks: - install: null - ceph: log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - rep_lost_unfound_delete: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T08:12:57.108 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa; will attempt to use it 2024-04-22T08:12:57.109 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa/tasks 2024-04-22T08:12:57.109 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T08:12:57.110 INFO:teuthology.task.internal:Checking packages... 2024-04-22T08:12:57.135 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '646b8c7d293d8ec31a40c69f7b3237b19c1be24d' 2024-04-22T08:12:57.135 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T08:12:57.135 INFO:teuthology.packaging:ref: None 2024-04-22T08:12:57.135 INFO:teuthology.packaging:tag: None 2024-04-22T08:12:57.135 INFO:teuthology.packaging:branch: reef 2024-04-22T08:12:57.135 INFO:teuthology.packaging:sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:12:57.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-04-22T08:12:57.498 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1237-g646b8c7d-1jammy 2024-04-22T08:12:57.500 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T08:12:57.506 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T08:12:57.506 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T08:12:57.528 INFO:teuthology.task.internal:Saving configuration 2024-04-22T08:12:57.542 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T08:12:57.547 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T08:12:57.581 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 08:05:16.160988', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINandGHbpXLNaVd+C9uTJ4SSAohpfokHDn64PszcVSXa'} 2024-04-22T08:12:57.581 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T08:12:57.587 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T08:12:57.587 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T08:12:57.660 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f443e1cf280>, signals=[15]) 2024-04-22T08:12:57.660 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T08:12:57.666 INFO:teuthology.task.internal:Opening connections... 2024-04-22T08:12:57.666 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:12:57.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:12:57.756 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T08:12:57.764 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-04-22T08:12:57.890 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-04-22T08:12:57.890 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="Ubuntu" 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="22.04" 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_CODENAME=jammy 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:ID=ubuntu 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE=debian 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-22T08:12:57.936 INFO:teuthology.orchestra.run.smithi049.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-22T08:12:57.937 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-22T08:12:57.937 INFO:teuthology.orchestra.run.smithi049.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-22T08:12:57.937 INFO:teuthology.orchestra.run.smithi049.stdout:UBUNTU_CODENAME=jammy 2024-04-22T08:12:57.937 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-04-22T08:12:57.966 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T08:12:57.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T08:12:57.979 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T08:12:57.979 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-04-22T08:12:57.983 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T08:12:57.989 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T08:12:57.989 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-04-22T08:12:58.033 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T08:12:58.075 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T08:12:58.088 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T08:12:58.088 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T08:12:58.089 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T08:12:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:12:58.094 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-04-22T08:12:58.140 INFO:teuthology.orchestra.run.smithi049.stdout:5.15.0-56-generic 2024-04-22T08:12:58.140 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.15.0-56-generic 2024-04-22T08:12:58.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-get clean 2024-04-22T08:12:58.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-get update 2024-04-22T08:12:58.521 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T08:12:58.521 INFO:teuthology.orchestra.run.smithi049.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-22T08:12:58.578 INFO:teuthology.orchestra.run.smithi049.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-22T08:12:58.929 INFO:teuthology.orchestra.run.smithi049.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-22T08:12:58.942 INFO:teuthology.orchestra.run.smithi049.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [450 kB] 2024-04-22T08:12:59.067 INFO:teuthology.orchestra.run.smithi049.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-22T08:12:59.235 INFO:teuthology.orchestra.run.smithi049.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,381 kB] 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi049.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi049.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [242 kB] 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi049.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi049.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi049.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi049.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,744 kB] 2024-04-22T08:12:59.430 INFO:teuthology.orchestra.run.smithi049.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-22T08:12:59.432 INFO:teuthology.orchestra.run.smithi049.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-22T08:12:59.494 INFO:teuthology.orchestra.run.smithi049.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-22T08:12:59.495 INFO:teuthology.orchestra.run.smithi049.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-22T08:12:59.499 INFO:teuthology.orchestra.run.smithi049.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [294 kB] 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi049.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi049.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi049.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi049.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi049.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-22T08:13:00.121 INFO:teuthology.orchestra.run.smithi049.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-22T08:13:00.127 INFO:teuthology.orchestra.run.smithi049.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-22T08:13:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-22T08:13:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-22T08:13:00.144 INFO:teuthology.orchestra.run.smithi049.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-22T08:13:00.275 INFO:teuthology.orchestra.run.smithi049.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-22T08:13:00.386 INFO:teuthology.orchestra.run.smithi049.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-22T08:13:00.406 INFO:teuthology.orchestra.run.smithi049.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-22T08:13:00.408 INFO:teuthology.orchestra.run.smithi049.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-22T08:13:00.408 INFO:teuthology.orchestra.run.smithi049.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-22T08:13:00.411 INFO:teuthology.orchestra.run.smithi049.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-22T08:13:00.412 INFO:teuthology.orchestra.run.smithi049.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-22T08:13:00.412 INFO:teuthology.orchestra.run.smithi049.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-22T08:13:00.417 INFO:teuthology.orchestra.run.smithi049.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-22T08:13:00.421 INFO:teuthology.orchestra.run.smithi049.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-22T08:13:00.586 INFO:teuthology.orchestra.run.smithi049.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-22T08:13:00.668 INFO:teuthology.orchestra.run.smithi049.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-22T08:13:00.754 INFO:teuthology.orchestra.run.smithi049.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-22T08:13:00.832 INFO:teuthology.orchestra.run.smithi049.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-22T08:13:00.869 INFO:teuthology.orchestra.run.smithi049.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-22T08:13:05.380 INFO:teuthology.orchestra.run.smithi049.stdout:Fetched 13.1 MB in 3s (5,145 kB/s) 2024-04-22T08:13:06.786 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:13:06.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T08:13:06.855 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:13:07.036 INFO:teuthology.orchestra.run.smithi049.stdout:Building dependency tree... 2024-04-22T08:13:07.037 INFO:teuthology.orchestra.run.smithi049.stdout:Reading state information... 2024-04-22T08:13:07.197 INFO:teuthology.orchestra.run.smithi049.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T08:13:07.198 INFO:teuthology.orchestra.run.smithi049.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T08:13:07.198 INFO:teuthology.orchestra.run.smithi049.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T08:13:07.198 INFO:teuthology.orchestra.run.smithi049.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T08:13:07.198 INFO:teuthology.orchestra.run.smithi049.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-22T08:13:07.198 INFO:teuthology.orchestra.run.smithi049.stdout: python2.7-minimal 2024-04-22T08:13:07.199 INFO:teuthology.orchestra.run.smithi049.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T08:13:07.199 INFO:teuthology.orchestra.run.smithi049.stdout:The following additional packages will be installed: 2024-04-22T08:13:07.199 INFO:teuthology.orchestra.run.smithi049.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T08:13:07.200 INFO:teuthology.orchestra.run.smithi049.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-22T08:13:07.200 INFO:teuthology.orchestra.run.smithi049.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-22T08:13:07.201 INFO:teuthology.orchestra.run.smithi049.stdout:Suggested packages: 2024-04-22T08:13:07.201 INFO:teuthology.orchestra.run.smithi049.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-22T08:13:07.201 INFO:teuthology.orchestra.run.smithi049.stdout:Recommended packages: 2024-04-22T08:13:07.201 INFO:teuthology.orchestra.run.smithi049.stdout: thermald 2024-04-22T08:13:07.243 INFO:teuthology.orchestra.run.smithi049.stdout:The following NEW packages will be installed: 2024-04-22T08:13:07.243 INFO:teuthology.orchestra.run.smithi049.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-22T08:13:07.243 INFO:teuthology.orchestra.run.smithi049.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-22T08:13:07.243 INFO:teuthology.orchestra.run.smithi049.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-22T08:13:07.244 INFO:teuthology.orchestra.run.smithi049.stdout:The following packages will be upgraded: 2024-04-22T08:13:07.245 INFO:teuthology.orchestra.run.smithi049.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-22T08:13:07.424 INFO:teuthology.orchestra.run.smithi049.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T08:13:07.424 INFO:teuthology.orchestra.run.smithi049.stdout:Need to get 113 MB of archives. 2024-04-22T08:13:07.424 INFO:teuthology.orchestra.run.smithi049.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-22T08:13:07.424 INFO:teuthology.orchestra.run.smithi049.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-22T08:13:08.838 INFO:teuthology.orchestra.run.smithi049.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-22T08:13:09.191 INFO:teuthology.orchestra.run.smithi049.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-22T08:13:11.388 INFO:teuthology.orchestra.run.smithi049.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-22T08:13:11.388 INFO:teuthology.orchestra.run.smithi049.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-22T08:13:11.388 INFO:teuthology.orchestra.run.smithi049.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-22T08:13:12.317 INFO:teuthology.orchestra.run.smithi049.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-22T08:13:12.411 INFO:teuthology.orchestra.run.smithi049.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-22T08:13:12.823 INFO:teuthology.orchestra.run.smithi049.stdout:Fetched 113 MB in 5s (22.0 MB/s) 2024-04-22T08:13:12.985 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-22T08:13:15.000 INFO:teuthology.orchestra.run.smithi049.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-22T08:13:15.004 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T08:13:15.066 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-22T08:13:17.173 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T08:13:17.267 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:17.720 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-22T08:13:17.741 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T08:13:17.782 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:24.217 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T08:13:24.334 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T08:13:24.692 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T08:13:24.804 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T08:13:25.073 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-22T08:13:25.084 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-22T08:13:25.123 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T08:13:30.912 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-22T08:13:30.949 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-22T08:13:30.987 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:33.741 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-22T08:13:34.315 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-22T08:13:34.940 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-22T08:13:35.061 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:35.190 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-22T08:13:35.318 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:36.649 INFO:teuthology.orchestra.run.smithi049.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-22T08:13:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-22T08:13:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-22T08:13:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-22T08:13:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:38.230 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:39.639 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-22T08:13:40.349 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-22T08:13:40.717 INFO:teuthology.orchestra.run.smithi049.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-22T08:13:40.790 INFO:teuthology.orchestra.run.smithi049.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-22T08:13:40.790 INFO:teuthology.orchestra.run.smithi049.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-22T08:13:55.146 INFO:teuthology.orchestra.run.smithi049.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-22T08:13:55.146 INFO:teuthology.orchestra.run.smithi049.stdout:Sourcing file `/etc/default/grub' 2024-04-22T08:13:55.153 INFO:teuthology.orchestra.run.smithi049.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T08:13:55.168 INFO:teuthology.orchestra.run.smithi049.stdout:Generating grub configuration file ... 2024-04-22T08:13:55.409 INFO:teuthology.orchestra.run.smithi049.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T08:13:55.424 INFO:teuthology.orchestra.run.smithi049.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T08:13:55.686 INFO:teuthology.orchestra.run.smithi049.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T08:13:55.692 INFO:teuthology.orchestra.run.smithi049.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T08:13:55.759 INFO:teuthology.orchestra.run.smithi049.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T08:13:55.765 INFO:teuthology.orchestra.run.smithi049.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T08:13:55.919 INFO:teuthology.orchestra.run.smithi049.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T08:13:55.920 INFO:teuthology.orchestra.run.smithi049.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T08:13:55.920 INFO:teuthology.orchestra.run.smithi049.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T08:13:55.963 INFO:teuthology.orchestra.run.smithi049.stdout:done 2024-04-22T08:13:56.412 DEBUG:teuthology.orchestra.run.smithi049:> dpkg -s linux-image-generic 2024-04-22T08:13:56.432 INFO:teuthology.orchestra.run.smithi049.stdout:Package: linux-image-generic 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Status: install ok installed 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Priority: optional 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Section: kernel 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Installed-Size: 21 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: amd64 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Source: linux-meta 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Version: 5.15.0.105.102 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Recommends: thermald 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:Description: Generic Linux kernel image 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: This package will always depend on the latest generic kernel image 2024-04-22T08:13:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: available. 2024-04-22T08:13:56.434 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-22T08:13:56.434 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-22T08:13:56.434 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-22T08:13:56.434 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-22T08:13:56.434 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-04-22T08:13:56.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-get clean 2024-04-22T08:13:56.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-get update 2024-04-22T08:13:56.730 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T08:13:56.730 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T08:13:56.733 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T08:13:56.753 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T08:13:58.141 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:13:58.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-22T08:13:58.223 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:13:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:Building dependency tree... 2024-04-22T08:13:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:Reading state information... 2024-04-22T08:13:58.623 INFO:teuthology.orchestra.run.smithi049.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-22T08:13:58.624 INFO:teuthology.orchestra.run.smithi049.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T08:13:58.624 INFO:teuthology.orchestra.run.smithi049.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T08:13:58.624 INFO:teuthology.orchestra.run.smithi049.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T08:13:58.624 INFO:teuthology.orchestra.run.smithi049.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T08:13:58.624 INFO:teuthology.orchestra.run.smithi049.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T08:13:58.625 INFO:teuthology.orchestra.run.smithi049.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T08:13:58.625 INFO:teuthology.orchestra.run.smithi049.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T08:13:58.665 INFO:teuthology.orchestra.run.smithi049.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-22T08:13:58.667 DEBUG:teuthology.orchestra.run.smithi049:> dpkg -s linux-image-generic 2024-04-22T08:13:58.680 INFO:teuthology.orchestra.run.smithi049.stdout:Package: linux-image-generic 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Status: install ok installed 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Priority: optional 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Section: kernel 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Installed-Size: 21 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Maintainer: Ubuntu Kernel Team 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: amd64 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Source: linux-meta 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Version: 5.15.0.105.102 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-22T08:13:58.681 INFO:teuthology.orchestra.run.smithi049.stdout:Recommends: thermald 2024-04-22T08:13:58.682 INFO:teuthology.orchestra.run.smithi049.stdout:Description: Generic Linux kernel image 2024-04-22T08:13:58.682 INFO:teuthology.orchestra.run.smithi049.stdout: This package will always depend on the latest generic kernel image 2024-04-22T08:13:58.682 INFO:teuthology.orchestra.run.smithi049.stdout: available. 2024-04-22T08:13:58.682 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-22T08:13:58.727 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.SVrso8xvmN 2024-04-22T08:13:58.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub/grub.cfg /tmp/tmp.SVrso8xvmN 2024-04-22T08:13:58.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.SVrso8xvmN 2024-04-22T08:13:58.897 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.SVrso8xvmN is 10KB 2024-04-22T08:13:58.913 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.SVrso8xvmN 2024-04-22T08:13:58.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-22T08:13:58.975 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:13:58.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-22T08:13:58.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-22T08:13:59.051 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-22T08:13:59.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo update-grub 2024-04-22T08:14:00.289 INFO:teuthology.orchestra.run.smithi049.stderr:Sourcing file `/etc/default/grub' 2024-04-22T08:14:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-22T08:14:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-04-22T08:14:00.463 INFO:teuthology.orchestra.run.smithi049.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-22T08:14:00.471 INFO:teuthology.orchestra.run.smithi049.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-22T08:14:00.686 INFO:teuthology.orchestra.run.smithi049.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-22T08:14:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-22T08:14:00.732 INFO:teuthology.orchestra.run.smithi049.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-22T08:14:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-22T08:14:00.870 INFO:teuthology.orchestra.run.smithi049.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-22T08:14:00.870 INFO:teuthology.orchestra.run.smithi049.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-22T08:14:00.870 INFO:teuthology.orchestra.run.smithi049.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-22T08:14:00.881 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-04-22T08:14:00.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-04-22T08:14:30.926 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-22T08:14:30.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:14:30.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:15:05.503 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-22T08:15:14.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:15:14.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:15:17.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-22T08:15:29.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:15:29.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:15:32.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-22T08:15:47.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:15:47.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:15:50.786 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-22T08:16:08.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:16:08.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:16:27.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-22T08:16:48.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:16:48.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:16:48.535 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-22T08:16:49.052 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:16:49.052 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-22T08:16:49.052 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-04-22T08:16:49.099 INFO:teuthology.orchestra.run.smithi049.stdout:5.15.0-105-generic 2024-04-22T08:16:49.099 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-22T08:16:49.099 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-22T08:16:49.099 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-22T08:16:50.100 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T08:16:50.101 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T08:16:50.156 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-04-22T08:16:50.183 DEBUG:teuthology.parallel:result is None 2024-04-22T08:16:50.184 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T08:16:50.190 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T08:16:50.191 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T08:16:50.196 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T08:16:50.203 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T08:16:50.210 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T08:16:50.210 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T08:16:50.262 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T08:16:50.269 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T08:16:50.269 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T08:16:50.325 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:16:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:16:50.336 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T08:16:50.343 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T08:16:50.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T08:16:50.388 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T08:16:50.397 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T08:16:50.397 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T08:16:50.434 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T08:16:50.482 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T08:16:50.530 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:16:50.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T08:16:50.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-04-22T08:16:50.668 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T08:16:50.676 INFO:teuthology.task.internal:Starting timer... 2024-04-22T08:16:50.676 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T08:16:50.685 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T08:16:50.695 DEBUG:teuthology.task.selinux:Excluding smithi049: OS 'ubuntu' does not support SELinux 2024-04-22T08:16:50.695 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T08:16:50.695 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-22T08:16:50.695 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T08:16:50.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T08:16:50.704 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T08:16:50.723 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T08:16:50.724 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T08:23:54.759 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2024-04-22T08:23:54.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:23:54.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:23:54.858 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-22T08:23:54.938 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T08:23:54.938 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T08:23:54.949 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T08:23:54.950 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T08:23:54.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:23:55.005 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-22T08:23:55.005 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Command line: ntpd -gq 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: ---------------------------------------------------- 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: ntp-4 is maintained by Network Time Foundation, 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: corporation. Support and training for ntp-4 are 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: available at https://www.nwtime.org/support 2024-04-22T08:23:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: ---------------------------------------------------- 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: proto: precision = 0.045 usec (-24) 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: basedate set to 2022-02-04 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: gps base set to 2022-02-06 (week 2196) 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-22T08:23:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: restrict ::: KOD does nothing without LIMITED. 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen and drop on 0 v6wildcard [::]:123 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen normally on 2 lo 127.0.0.1:123 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen normally on 3 ens1f0 172.21.15.49:123 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen normally on 4 lo [::1]:123 2024-04-22T08:23:55.008 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1566%4]:123 2024-04-22T08:23:55.009 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:23:55 ntpd[17101]: Listening on routing socket on fd #22 for interface updates 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:23:56.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:56 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:23:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:23:57.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:57 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:23:58.006 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:23:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:58 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:23:59.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:59.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:23:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:23:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:23:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:23:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:23:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:23:59 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:24:00.006 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:00.006 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:00.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:00.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:00.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:24:00.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:00 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:24:01.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:01.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:01.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:01.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:01.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:24:01.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:01 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:24:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: can't open /var/log/ntpstats/rawstats.20240422: Permission denied 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: can't open /var/log/ntpstats/peerstats.20240422: Permission denied 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stdout:22 Apr 08:24:02 ntpd[17101]: ntpd: time slew -0.002237 s 2024-04-22T08:24:02.007 INFO:teuthology.orchestra.run.smithi049.stdout:ntpd: time slew -0.002237s 2024-04-22T08:24:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-22T08:24:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:22 Apr 08:24:02 ntpd[17101]: can't open /var/log/ntpstats/loopstats.20240422: Permission denied 2024-04-22T08:24:02.056 INFO:teuthology.orchestra.run.smithi049.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T08:24:02.056 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================== 2024-04-22T08:24:02.056 INFO:teuthology.orchestra.run.smithi049.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T08:24:02.056 INFO:teuthology.orchestra.run.smithi049.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T08:24:02.056 INFO:teuthology.orchestra.run.smithi049.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T08:24:02.057 INFO:teuthology.orchestra.run.smithi049.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-22T08:24:02.057 INFO:teuthology.run_tasks:Running task install... 2024-04-22T08:24:02.096 DEBUG:teuthology.task.install:project ceph 2024-04-22T08:24:02.096 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}} 2024-04-22T08:24:02.096 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'} 2024-04-22T08:24:02.096 INFO:teuthology.task.install:Using flavor: default 2024-04-22T08:24:02.102 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T08:24:02.102 INFO:teuthology.task.install:extra packages: [] 2024-04-22T08:24:02.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-key list | grep Ceph 2024-04-22T08:24:02.163 INFO:teuthology.orchestra.run.smithi049.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-22T08:24:02.193 INFO:teuthology.orchestra.run.smithi049.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-22T08:24:02.193 INFO:teuthology.orchestra.run.smithi049.stdout:uid [ unknown] Ceph.com (release key) 2024-04-22T08:24:02.193 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-22T08:24:02.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:24:02.345 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default/ 2024-04-22T08:24:02.346 INFO:teuthology.task.install.deb:Package version is 18.2.2-1237-g646b8c7d-1jammy 2024-04-22T08:24:02.468 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:24:02.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-22T08:24:02.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo apt-get update 2024-04-22T08:24:02.779 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-22T08:24:02.779 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-22T08:24:02.783 INFO:teuthology.orchestra.run.smithi049.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy InRelease 2024-04-22T08:24:02.820 INFO:teuthology.orchestra.run.smithi049.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-22T08:24:02.852 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-22T08:24:02.857 INFO:teuthology.orchestra.run.smithi049.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-22T08:24:02.894 INFO:teuthology.orchestra.run.smithi049.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2024-04-22T08:24:02.933 INFO:teuthology.orchestra.run.smithi049.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-22T08:24:02.938 INFO:teuthology.orchestra.run.smithi049.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,429 B] 2024-04-22T08:24:03.148 INFO:teuthology.orchestra.run.smithi049.stdout:Fetched 28.6 kB in 0s (58.8 kB/s) 2024-04-22T08:24:04.125 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:24:04.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1237-g646b8c7d-1jammy cephadm=18.2.2-1237-g646b8c7d-1jammy ceph-mds=18.2.2-1237-g646b8c7d-1jammy ceph-mgr=18.2.2-1237-g646b8c7d-1jammy ceph-common=18.2.2-1237-g646b8c7d-1jammy ceph-fuse=18.2.2-1237-g646b8c7d-1jammy ceph-test=18.2.2-1237-g646b8c7d-1jammy ceph-volume=18.2.2-1237-g646b8c7d-1jammy radosgw=18.2.2-1237-g646b8c7d-1jammy python3-rados=18.2.2-1237-g646b8c7d-1jammy python3-rgw=18.2.2-1237-g646b8c7d-1jammy python3-cephfs=18.2.2-1237-g646b8c7d-1jammy python3-rbd=18.2.2-1237-g646b8c7d-1jammy libcephfs2=18.2.2-1237-g646b8c7d-1jammy libcephfs-dev=18.2.2-1237-g646b8c7d-1jammy librados2=18.2.2-1237-g646b8c7d-1jammy librbd1=18.2.2-1237-g646b8c7d-1jammy rbd-fuse=18.2.2-1237-g646b8c7d-1jammy 2024-04-22T08:24:04.204 INFO:teuthology.orchestra.run.smithi049.stdout:Reading package lists... 2024-04-22T08:24:04.408 INFO:teuthology.orchestra.run.smithi049.stdout:Building dependency tree... 2024-04-22T08:24:04.409 INFO:teuthology.orchestra.run.smithi049.stdout:Reading state information... 2024-04-22T08:24:04.608 INFO:teuthology.orchestra.run.smithi049.stdout:The following packages were automatically installed and are no longer required: 2024-04-22T08:24:04.608 INFO:teuthology.orchestra.run.smithi049.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-22T08:24:04.608 INFO:teuthology.orchestra.run.smithi049.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-22T08:24:04.608 INFO:teuthology.orchestra.run.smithi049.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-22T08:24:04.609 INFO:teuthology.orchestra.run.smithi049.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-22T08:24:04.609 INFO:teuthology.orchestra.run.smithi049.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-22T08:24:04.609 INFO:teuthology.orchestra.run.smithi049.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-22T08:24:04.609 INFO:teuthology.orchestra.run.smithi049.stdout:The following additional packages will be installed: 2024-04-22T08:24:04.609 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-22T08:24:04.610 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-22T08:24:04.610 INFO:teuthology.orchestra.run.smithi049.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-04-22T08:24:04.610 INFO:teuthology.orchestra.run.smithi049.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-22T08:24:04.611 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-04-22T08:24:04.612 INFO:teuthology.orchestra.run.smithi049.stdout:Suggested packages: 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: python3-watchdog 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout:Recommended packages: 2024-04-22T08:24:04.613 INFO:teuthology.orchestra.run.smithi049.stdout: btrfs-tools 2024-04-22T08:24:04.717 INFO:teuthology.orchestra.run.smithi049.stdout:The following NEW packages will be installed: 2024-04-22T08:24:04.717 INFO:teuthology.orchestra.run.smithi049.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-22T08:24:04.717 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-22T08:24:04.717 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-22T08:24:04.718 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-22T08:24:04.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-22T08:24:04.720 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-22T08:24:04.720 INFO:teuthology.orchestra.run.smithi049.stdout: socat xmlstarlet 2024-04-22T08:24:04.784 INFO:teuthology.orchestra.run.smithi049.stdout:0 upgraded, 109 newly installed, 0 to remove and 302 not upgraded. 2024-04-22T08:24:04.784 INFO:teuthology.orchestra.run.smithi049.stdout:Need to get 162 MB of archives. 2024-04-22T08:24:04.784 INFO:teuthology.orchestra.run.smithi049.stdout:After this operation, 743 MB of additional disk space will be used. 2024-04-22T08:24:04.785 INFO:teuthology.orchestra.run.smithi049.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-04-22T08:24:04.850 INFO:teuthology.orchestra.run.smithi049.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-22T08:24:04.861 INFO:teuthology.orchestra.run.smithi049.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1237-g646b8c7d-1jammy [3,440 kB] 2024-04-22T08:24:04.892 INFO:teuthology.orchestra.run.smithi049.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-22T08:24:04.895 INFO:teuthology.orchestra.run.smithi049.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-22T08:24:04.913 INFO:teuthology.orchestra.run.smithi049.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-22T08:24:04.982 INFO:teuthology.orchestra.run.smithi049.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-22T08:24:04.988 INFO:teuthology.orchestra.run.smithi049.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-22T08:24:05.001 INFO:teuthology.orchestra.run.smithi049.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-22T08:24:05.005 INFO:teuthology.orchestra.run.smithi049.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-22T08:24:05.006 INFO:teuthology.orchestra.run.smithi049.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-22T08:24:05.007 INFO:teuthology.orchestra.run.smithi049.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-22T08:24:05.015 INFO:teuthology.orchestra.run.smithi049.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-22T08:24:05.016 INFO:teuthology.orchestra.run.smithi049.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-22T08:24:05.017 INFO:teuthology.orchestra.run.smithi049.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-22T08:24:05.018 INFO:teuthology.orchestra.run.smithi049.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-22T08:24:05.027 INFO:teuthology.orchestra.run.smithi049.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-22T08:24:05.027 INFO:teuthology.orchestra.run.smithi049.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-22T08:24:05.027 INFO:teuthology.orchestra.run.smithi049.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-22T08:24:05.046 INFO:teuthology.orchestra.run.smithi049.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-22T08:24:05.047 INFO:teuthology.orchestra.run.smithi049.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-22T08:24:05.047 INFO:teuthology.orchestra.run.smithi049.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-22T08:24:05.050 INFO:teuthology.orchestra.run.smithi049.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-22T08:24:05.051 INFO:teuthology.orchestra.run.smithi049.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-22T08:24:05.051 INFO:teuthology.orchestra.run.smithi049.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-22T08:24:05.052 INFO:teuthology.orchestra.run.smithi049.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-22T08:24:05.063 INFO:teuthology.orchestra.run.smithi049.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1237-g646b8c7d-1jammy [3,106 kB] 2024-04-22T08:24:05.065 INFO:teuthology.orchestra.run.smithi049.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-22T08:24:05.066 INFO:teuthology.orchestra.run.smithi049.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-22T08:24:05.067 INFO:teuthology.orchestra.run.smithi049.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-22T08:24:05.085 INFO:teuthology.orchestra.run.smithi049.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-22T08:24:05.086 INFO:teuthology.orchestra.run.smithi049.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-22T08:24:05.087 INFO:teuthology.orchestra.run.smithi049.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-22T08:24:05.095 INFO:teuthology.orchestra.run.smithi049.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-22T08:24:05.096 INFO:teuthology.orchestra.run.smithi049.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-22T08:24:05.097 INFO:teuthology.orchestra.run.smithi049.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-22T08:24:05.098 INFO:teuthology.orchestra.run.smithi049.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-22T08:24:05.104 INFO:teuthology.orchestra.run.smithi049.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-22T08:24:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-22T08:24:05.108 INFO:teuthology.orchestra.run.smithi049.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-22T08:24:05.125 INFO:teuthology.orchestra.run.smithi049.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-22T08:24:05.125 INFO:teuthology.orchestra.run.smithi049.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-22T08:24:05.128 INFO:teuthology.orchestra.run.smithi049.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-22T08:24:05.129 INFO:teuthology.orchestra.run.smithi049.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-22T08:24:05.133 INFO:teuthology.orchestra.run.smithi049.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-22T08:24:05.157 INFO:teuthology.orchestra.run.smithi049.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1237-g646b8c7d-1jammy [680 kB] 2024-04-22T08:24:05.179 INFO:teuthology.orchestra.run.smithi049.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1237-g646b8c7d-1jammy [356 kB] 2024-04-22T08:24:05.190 INFO:teuthology.orchestra.run.smithi049.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-22T08:24:05.190 INFO:teuthology.orchestra.run.smithi049.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1237-g646b8c7d-1jammy [32.8 kB] 2024-04-22T08:24:05.190 INFO:teuthology.orchestra.run.smithi049.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-22T08:24:05.191 INFO:teuthology.orchestra.run.smithi049.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1237-g646b8c7d-1jammy [180 kB] 2024-04-22T08:24:05.191 INFO:teuthology.orchestra.run.smithi049.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-22T08:24:05.192 INFO:teuthology.orchestra.run.smithi049.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-22T08:24:05.192 INFO:teuthology.orchestra.run.smithi049.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-22T08:24:05.194 INFO:teuthology.orchestra.run.smithi049.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-22T08:24:05.194 INFO:teuthology.orchestra.run.smithi049.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-22T08:24:05.194 INFO:teuthology.orchestra.run.smithi049.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-22T08:24:05.196 INFO:teuthology.orchestra.run.smithi049.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-22T08:24:05.205 INFO:teuthology.orchestra.run.smithi049.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1237-g646b8c7d-1jammy [63.7 kB] 2024-04-22T08:24:05.206 INFO:teuthology.orchestra.run.smithi049.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1237-g646b8c7d-1jammy [330 kB] 2024-04-22T08:24:05.211 INFO:teuthology.orchestra.run.smithi049.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-22T08:24:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1237-g646b8c7d-1jammy [5,891 kB] 2024-04-22T08:24:05.223 INFO:teuthology.orchestra.run.smithi049.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-22T08:24:05.242 INFO:teuthology.orchestra.run.smithi049.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-22T08:24:05.244 INFO:teuthology.orchestra.run.smithi049.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-22T08:24:05.244 INFO:teuthology.orchestra.run.smithi049.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-22T08:24:05.263 INFO:teuthology.orchestra.run.smithi049.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-22T08:24:05.263 INFO:teuthology.orchestra.run.smithi049.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-22T08:24:05.263 INFO:teuthology.orchestra.run.smithi049.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-22T08:24:05.264 INFO:teuthology.orchestra.run.smithi049.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-22T08:24:05.264 INFO:teuthology.orchestra.run.smithi049.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-22T08:24:05.265 INFO:teuthology.orchestra.run.smithi049.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-22T08:24:05.284 INFO:teuthology.orchestra.run.smithi049.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-22T08:24:05.284 INFO:teuthology.orchestra.run.smithi049.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-22T08:24:05.288 INFO:teuthology.orchestra.run.smithi049.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-22T08:24:05.289 INFO:teuthology.orchestra.run.smithi049.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-22T08:24:05.291 INFO:teuthology.orchestra.run.smithi049.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-22T08:24:05.303 INFO:teuthology.orchestra.run.smithi049.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-22T08:24:05.307 INFO:teuthology.orchestra.run.smithi049.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-22T08:24:05.309 INFO:teuthology.orchestra.run.smithi049.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-22T08:24:05.313 INFO:teuthology.orchestra.run.smithi049.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-22T08:24:05.316 INFO:teuthology.orchestra.run.smithi049.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-22T08:24:05.323 INFO:teuthology.orchestra.run.smithi049.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-22T08:24:05.324 INFO:teuthology.orchestra.run.smithi049.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-22T08:24:05.331 INFO:teuthology.orchestra.run.smithi049.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-22T08:24:05.332 INFO:teuthology.orchestra.run.smithi049.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-22T08:24:05.332 INFO:teuthology.orchestra.run.smithi049.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-22T08:24:05.405 INFO:teuthology.orchestra.run.smithi049.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1237-g646b8c7d-1jammy [112 kB] 2024-04-22T08:24:05.408 INFO:teuthology.orchestra.run.smithi049.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1237-g646b8c7d-1jammy [441 kB] 2024-04-22T08:24:05.422 INFO:teuthology.orchestra.run.smithi049.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1237-g646b8c7d-1jammy [21.9 MB] 2024-04-22T08:24:06.234 INFO:teuthology.orchestra.run.smithi049.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1237-g646b8c7d-1jammy [5,002 kB] 2024-04-22T08:24:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1237-g646b8c7d-1jammy [241 kB] 2024-04-22T08:24:06.321 INFO:teuthology.orchestra.run.smithi049.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1237-g646b8c7d-1jammy [130 kB] 2024-04-22T08:24:06.325 INFO:teuthology.orchestra.run.smithi049.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1237-g646b8c7d-1jammy [1,490 kB] 2024-04-22T08:24:06.374 INFO:teuthology.orchestra.run.smithi049.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1237-g646b8c7d-1jammy [6,079 kB] 2024-04-22T08:24:06.575 INFO:teuthology.orchestra.run.smithi049.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1237-g646b8c7d-1jammy [22.3 MB] 2024-04-22T08:24:07.409 INFO:teuthology.orchestra.run.smithi049.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1237-g646b8c7d-1jammy [14.1 kB] 2024-04-22T08:24:07.409 INFO:teuthology.orchestra.run.smithi049.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1237-g646b8c7d-1jammy [831 kB] 2024-04-22T08:24:07.425 INFO:teuthology.orchestra.run.smithi049.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1237-g646b8c7d-1jammy [2,154 kB] 2024-04-22T08:24:07.465 INFO:teuthology.orchestra.run.smithi049.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1237-g646b8c7d-1jammy [238 kB] 2024-04-22T08:24:07.468 INFO:teuthology.orchestra.run.smithi049.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1237-g646b8c7d-1jammy [144 kB] 2024-04-22T08:24:07.470 INFO:teuthology.orchestra.run.smithi049.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1237-g646b8c7d-1jammy [2,238 kB] 2024-04-22T08:24:07.495 INFO:teuthology.orchestra.run.smithi049.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1237-g646b8c7d-1jammy [8,625 kB] 2024-04-22T08:24:07.774 INFO:teuthology.orchestra.run.smithi049.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1237-g646b8c7d-1jammy [14.3 kB] 2024-04-22T08:24:07.775 INFO:teuthology.orchestra.run.smithi049.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1237-g646b8c7d-1jammy [42.6 MB] 2024-04-22T08:24:09.350 INFO:teuthology.orchestra.run.smithi049.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1237-g646b8c7d-1jammy [121 kB] 2024-04-22T08:24:09.353 INFO:teuthology.orchestra.run.smithi049.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1237-g646b8c7d-1jammy [38.3 kB] 2024-04-22T08:24:09.354 INFO:teuthology.orchestra.run.smithi049.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1237-g646b8c7d-1jammy [11.6 MB] 2024-04-22T08:24:09.621 INFO:teuthology.orchestra.run.smithi049.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1237-g646b8c7d-1jammy [93.0 kB] 2024-04-22T08:24:10.262 INFO:teuthology.orchestra.run.smithi049.stdout:Fetched 162 MB in 5s (32.8 MB/s) 2024-04-22T08:24:10.396 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libfmt8:amd64. 2024-04-22T08:24:10.487 INFO:teuthology.orchestra.run.smithi049.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-22T08:24:10.493 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-04-22T08:24:10.578 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-22T08:24:10.914 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-22T08:24:10.941 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-22T08:24:10.981 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T08:24:11.326 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-22T08:24:11.353 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-22T08:24:11.393 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T08:24:11.695 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-22T08:24:11.723 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-22T08:24:11.762 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T08:24:12.241 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-22T08:24:12.268 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T08:24:12.316 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:24:12.702 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-22T08:24:12.729 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T08:24:12.769 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:24:13.105 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-22T08:24:13.132 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-22T08:24:13.171 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:24:13.541 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-22T08:24:13.568 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-22T08:24:13.608 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T08:24:13.944 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package librados2. 2024-04-22T08:24:13.971 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../008-librados2_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:14.010 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking librados2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:14.455 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package librbd1. 2024-04-22T08:24:14.482 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../009-librbd1_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:14.521 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking librbd1 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:14.975 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libcephfs2. 2024-04-22T08:24:15.001 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:15.041 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libcephfs2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:15.319 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-rados. 2024-04-22T08:24:15.345 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:15.385 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-rados (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:15.679 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-22T08:24:15.706 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:15.746 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-ceph-argparse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:15.982 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-cephfs. 2024-04-22T08:24:16.008 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:16.049 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-cephfs (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:16.360 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-22T08:24:16.386 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:16.426 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-ceph-common (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:16.771 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-22T08:24:16.797 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-22T08:24:16.837 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T08:24:17.098 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-prettytable. 2024-04-22T08:24:17.125 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-04-22T08:24:17.165 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-22T08:24:17.418 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-rbd. 2024-04-22T08:24:17.434 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:17.476 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-rbd (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:17.821 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-22T08:24:17.848 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-22T08:24:17.887 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T08:24:18.340 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package librgw2. 2024-04-22T08:24:18.367 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../019-librgw2_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:18.406 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking librgw2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:18.876 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-rgw. 2024-04-22T08:24:18.892 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:18.925 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-rgw (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:19.295 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-22T08:24:19.322 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-22T08:24:19.362 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T08:24:19.656 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libradosstriper1. 2024-04-22T08:24:19.683 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:19.722 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libradosstriper1 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:20.143 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-common. 2024-04-22T08:24:20.170 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:20.209 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-common (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:21.431 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-base. 2024-04-22T08:24:21.458 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:21.507 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-base (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:22.052 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-22T08:24:22.080 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-22T08:24:22.119 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-22T08:24:22.422 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-cheroot. 2024-04-22T08:24:22.449 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-22T08:24:22.488 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T08:24:22.842 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-22T08:24:22.869 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-22T08:24:22.908 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-22T08:24:23.186 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-22T08:24:23.213 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-22T08:24:23.252 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-22T08:24:23.813 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-22T08:24:23.841 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-22T08:24:23.921 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-22T08:24:24.225 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-tempora. 2024-04-22T08:24:24.252 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-04-22T08:24:24.291 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-22T08:24:24.586 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-portend. 2024-04-22T08:24:24.612 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-04-22T08:24:24.652 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-22T08:24:24.896 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-22T08:24:24.912 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-22T08:24:24.955 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-22T08:24:25.260 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-22T08:24:25.287 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-22T08:24:25.326 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-22T08:24:25.680 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-natsort. 2024-04-22T08:24:25.707 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-04-22T08:24:25.746 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-22T08:24:26.058 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-logutils. 2024-04-22T08:24:26.085 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-04-22T08:24:26.124 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-22T08:24:26.444 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-22T08:24:26.471 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-22T08:24:26.510 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-22T08:24:26.755 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-mako. 2024-04-22T08:24:26.782 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-22T08:24:26.813 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T08:24:27.124 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-22T08:24:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-22T08:24:27.191 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-22T08:24:27.502 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-22T08:24:27.529 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-22T08:24:27.569 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-22T08:24:27.872 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-webob. 2024-04-22T08:24:27.898 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-22T08:24:27.938 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-22T08:24:28.274 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-waitress. 2024-04-22T08:24:28.301 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-22T08:24:28.344 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T08:24:28.661 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-tempita. 2024-04-22T08:24:28.687 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-22T08:24:28.727 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T08:24:29.047 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-paste. 2024-04-22T08:24:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-22T08:24:29.113 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T08:24:29.441 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-22T08:24:29.457 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-22T08:24:29.499 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T08:24:29.836 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-22T08:24:29.863 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-22T08:24:29.902 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-22T08:24:30.205 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-webtest. 2024-04-22T08:24:30.233 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-04-22T08:24:30.272 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-22T08:24:30.583 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pecan. 2024-04-22T08:24:30.610 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-22T08:24:30.649 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T08:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-certifi. 2024-04-22T08:24:31.048 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-04-22T08:24:31.087 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-22T08:24:31.374 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-idna. 2024-04-22T08:24:31.390 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-04-22T08:24:31.423 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-22T08:24:31.743 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-urllib3. 2024-04-22T08:24:31.770 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-22T08:24:31.818 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T08:24:32.138 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-requests. 2024-04-22T08:24:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-22T08:24:32.206 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T08:24:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-22T08:24:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-22T08:24:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T08:24:32.927 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-22T08:24:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:32.985 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:33.366 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-22T08:24:33.393 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:33.726 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-openssl. 2024-04-22T08:24:33.753 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-04-22T08:24:33.792 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-22T08:24:34.296 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr. 2024-04-22T08:24:34.323 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:34.370 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:34.773 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mon. 2024-04-22T08:24:34.801 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:34.839 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mon (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-osd. 2024-04-22T08:24:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-osd (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:36.638 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph. 2024-04-22T08:24:36.654 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../059-ceph_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:36.696 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-fuse. 2024-04-22T08:24:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-fuse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:37.419 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mds. 2024-04-22T08:24:37.446 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:37.477 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mds (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:37.863 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package cephadm. 2024-04-22T08:24:37.890 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../062-cephadm_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:37.929 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking cephadm (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:38.992 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-22T08:24:39.019 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-22T08:24:39.058 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-22T08:24:39.361 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-22T08:24:39.377 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T08:24:39.411 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T08:24:40.441 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-babel. 2024-04-22T08:24:40.470 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-22T08:24:40.507 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T08:24:40.852 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-jinja2. 2024-04-22T08:24:40.879 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-22T08:24:40.918 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T08:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-22T08:24:41.299 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:41.338 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:41.683 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-22T08:24:41.711 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-22T08:24:41.750 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-22T08:24:42.019 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pluggy. 2024-04-22T08:24:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-22T08:24:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-22T08:24:42.347 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-py. 2024-04-22T08:24:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-04-22T08:24:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-22T08:24:42.742 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-22T08:24:42.770 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-22T08:24:42.809 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-22T08:24:43.120 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-toml. 2024-04-22T08:24:43.148 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-04-22T08:24:43.186 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-22T08:24:43.464 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pytest. 2024-04-22T08:24:43.492 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-22T08:24:43.531 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T08:24:43.918 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-22T08:24:43.946 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-04-22T08:24:43.984 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-22T08:24:44.270 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-routes. 2024-04-22T08:24:44.290 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-22T08:24:44.320 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T08:24:44.657 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-22T08:24:44.685 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:44.723 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:45.979 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-22T08:24:46.006 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-22T08:24:46.045 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T08:24:46.549 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-joblib. 2024-04-22T08:24:46.576 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-22T08:24:46.615 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T08:24:46.986 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-22T08:24:47.014 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-22T08:24:47.052 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-22T08:24:47.330 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-sklearn. 2024-04-22T08:24:47.357 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-22T08:24:47.388 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T08:24:48.122 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-22T08:24:48.150 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:48.188 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:49.158 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-cachetools. 2024-04-22T08:24:49.186 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-04-22T08:24:49.278 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-22T08:24:49.812 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-22T08:24:49.840 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-22T08:24:49.879 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-22T08:24:50.208 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-22T08:24:50.236 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-22T08:24:50.274 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T08:24:50.585 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-rsa. 2024-04-22T08:24:50.613 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-04-22T08:24:50.660 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-22T08:24:50.964 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-google-auth. 2024-04-22T08:24:50.991 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-04-22T08:24:51.030 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-22T08:24:51.392 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-22T08:24:51.419 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-22T08:24:51.458 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T08:24:51.769 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-websocket. 2024-04-22T08:24:51.797 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-04-22T08:24:51.836 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-22T08:24:52.148 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-22T08:24:52.175 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-22T08:24:52.214 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T08:24:52.911 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-22T08:24:52.940 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:52.977 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:53.305 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-22T08:24:53.333 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-22T08:24:53.372 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T08:24:53.717 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-22T08:24:53.745 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T08:24:53.783 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T08:24:54.002 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package jq. 2024-04-22T08:24:54.019 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-22T08:24:54.052 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-22T08:24:54.305 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package socat. 2024-04-22T08:24:54.322 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-22T08:24:54.363 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-22T08:24:54.658 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package xmlstarlet. 2024-04-22T08:24:54.675 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-22T08:24:54.708 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-22T08:24:55.104 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-test. 2024-04-22T08:24:55.121 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:55.162 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-test (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:57.117 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package ceph-volume. 2024-04-22T08:24:57.144 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1237-g646b8c7d-1jammy_all.deb ... 2024-04-22T08:24:57.182 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking ceph-volume (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:57.495 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-22T08:24:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:24:57.561 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libcephfs-dev (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:24:57.873 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package nvme-cli. 2024-04-22T08:24:57.901 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-22T08:24:57.931 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T08:24:58.328 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-22T08:24:58.356 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-22T08:24:58.403 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-22T08:24:58.898 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pastescript. 2024-04-22T08:24:58.926 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-04-22T08:24:58.972 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-22T08:24:59.326 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-psutil. 2024-04-22T08:24:59.354 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-22T08:24:59.400 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-22T08:24:59.987 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pygments. 2024-04-22T08:25:00.015 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-22T08:25:00.053 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T08:25:00.466 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-22T08:25:00.494 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-22T08:25:00.532 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-22T08:25:00.843 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package python3-simplejson. 2024-04-22T08:25:00.871 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-22T08:25:00.909 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-22T08:25:01.187 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-22T08:25:01.215 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-22T08:25:01.254 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-22T08:25:02.058 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package radosgw. 2024-04-22T08:25:02.075 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../107-radosgw_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:25:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking radosgw (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:02.886 INFO:teuthology.orchestra.run.smithi049.stdout:Selecting previously unselected package rbd-fuse. 2024-04-22T08:25:02.911 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1237-g646b8c7d-1jammy_amd64.deb ... 2024-04-22T08:25:02.944 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking rbd-fuse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:03.273 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-22T08:25:03.632 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-22T08:25:03.758 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-22T08:25:04.026 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-22T08:25:04.350 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-22T08:25:04.772 INFO:teuthology.orchestra.run.smithi049.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-22T08:25:04.877 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up cephadm (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:05.761 INFO:teuthology.orchestra.run.smithi049.stdout:Adding system user cephadm....done 2024-04-22T08:25:05.917 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-22T08:25:06.193 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-22T08:25:06.438 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-22T08:25:06.564 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-22T08:25:06.823 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-22T08:25:07.099 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-22T08:25:07.226 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-22T08:25:07.511 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-22T08:25:07.854 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-22T08:25:08.114 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-22T08:25:08.390 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-22T08:25:08.649 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-22T08:25:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-22T08:25:09.160 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-ceph-argparse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:09.428 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-22T08:25:09.562 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-22T08:25:09.821 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-22T08:25:09.948 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-22T08:25:10.324 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-22T08:25:10.449 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-22T08:25:10.717 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-22T08:25:10.968 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-22T08:25:11.269 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-22T08:25:11.980 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-22T08:25:12.239 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-22T08:25:12.366 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-22T08:25:12.491 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-22T08:25:12.809 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-22T08:25:13.186 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-22T08:25:13.437 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-22T08:25:13.822 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-22T08:25:14.123 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-22T08:25:14.508 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-22T08:25:14.759 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-22T08:25:14.859 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-ceph-common (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:15.361 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-idna (3.3-1) ... 2024-04-22T08:25:15.696 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-22T08:25:16.910 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:25:17.027 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-22T08:25:17.304 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-22T08:25:17.430 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-22T08:25:17.556 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-22T08:25:17.866 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-22T08:25:18.133 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-22T08:25:18.368 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:25:18.494 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-22T08:25:18.620 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-22T08:25:18.930 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-22T08:25:19.189 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-22T08:25:19.465 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-22T08:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-22T08:25:19.984 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-22T08:25:20.243 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-22T08:25:20.361 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-22T08:25:20.645 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-22T08:25:20.772 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-22T08:25:21.056 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-22T08:25:21.357 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-22T08:25:21.608 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-22T08:25:21.994 INFO:teuthology.orchestra.run.smithi049.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-22T08:25:22.253 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-22T08:25:22.379 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-22T08:25:22.772 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-22T08:25:23.032 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-22T08:25:23.291 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-22T08:25:23.417 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-22T08:25:23.751 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-22T08:25:24.094 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-22T08:25:24.379 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-22T08:25:24.505 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-22T08:25:24.789 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-22T08:25:25.048 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-22T08:25:25.558 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-22T08:25:25.926 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-22T08:25:26.211 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-22T08:25:26.572 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-22T08:25:26.673 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up librados2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:26.790 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up librgw2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:26.917 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libcephfs2 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:27.193 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libradosstriper1 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:27.388 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-22T08:25:27.674 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up librbd1 (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:27.791 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr-modules-core (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:27.892 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-fuse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:28.100 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T08:25:28.100 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-22T08:25:28.544 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up libcephfs-dev (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:28.670 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-rados (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:28.796 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-22T08:25:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-rbd (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:30.210 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up rbd-fuse (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:30.336 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-rgw (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:30.470 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up python3-cephfs (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:30.596 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-common (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:31.364 INFO:teuthology.orchestra.run.smithi049.stdout:Adding group ceph....done 2024-04-22T08:25:31.700 INFO:teuthology.orchestra.run.smithi049.stdout:Adding system user ceph....done 2024-04-22T08:25:31.868 INFO:teuthology.orchestra.run.smithi049.stdout:Setting system user ceph properties....done 2024-04-22T08:25:31.884 INFO:teuthology.orchestra.run.smithi049.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-22T08:25:32.028 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-22T08:25:32.372 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-22T08:25:32.823 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-test (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up radosgw (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:33.375 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:25:33.375 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:25:33.842 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-base (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:34.192 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-22T08:25:34.652 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mds (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T08:25:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-22T08:25:35.371 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T08:25:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-22T08:25:36.039 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-osd (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:36.347 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T08:25:36.347 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-22T08:25:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:36.925 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mon (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:37.295 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T08:25:37.295 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-22T08:25:37.719 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr-cephadm (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:37.887 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:38.013 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-mgr-dashboard (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:38.164 INFO:teuthology.orchestra.run.smithi049.stdout:Setting up ceph-volume (18.2.2-1237-g646b8c7d-1jammy) ... 2024-04-22T08:25:38.534 INFO:teuthology.orchestra.run.smithi049.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-22T08:25:39.669 INFO:teuthology.orchestra.run.smithi049.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-22T08:25:40.156 INFO:teuthology.orchestra.run.smithi049.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-22T08:25:40.162 DEBUG:teuthology.parallel:result is None 2024-04-22T08:25:40.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:25:40.389 DEBUG:teuthology.orchestra.run.smithi049:> dpkg-query -W -f '${Version}' ceph 2024-04-22T08:25:40.409 INFO:teuthology.orchestra.run.smithi049.stdout:18.2.2-1237-g646b8c7d-1jammy 2024-04-22T08:25:40.409 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1237-g646b8c7d-1jammy 2024-04-22T08:25:40.409 INFO:teuthology.task.install:The correct ceph version 18.2.2-1237-g646b8c7d-1jammy is installed. 2024-04-22T08:25:40.411 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T08:25:40.411 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:40.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T08:25:40.466 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T08:25:40.466 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:40.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T08:25:40.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T08:25:40.576 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T08:25:40.576 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:40.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T08:25:40.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T08:25:40.696 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T08:25:40.696 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:40.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T08:25:40.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T08:25:40.809 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T08:25:40.893 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T08:25:40.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2024-04-22T08:25:40.906 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T08:25:40.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T08:25:40.960 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T08:25:40.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T08:25:41.024 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T08:25:41.024 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-04-22T08:25:41.024 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875', 'branch': 'reef', 'description': 'rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7667875', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'mon_bind_msgr2': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'c3a345d31ea77c01771428ab80e1cf32c9b05875'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'seed': 7353, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'sleep_before_teardown': 0, 'subset': '3333/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'c3a345d31ea77c01771428ab80e1cf32c9b05875', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPXa6y+CGC715PpLZE+5qeK4KRv0t7J8H9pbJCwdRLylAubUnXtl8lrHFVEQjGHTCUNDdJIqkf3B6PV4qkgJsc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'cluster': 'ceph'}}, {'rep_lost_unfound_delete': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-21_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T08:25:41.025 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T08:25:41.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T08:25:41.120 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:41.121 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T08:25:41.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T08:25:41.169 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-04-22T08:25:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T08:25:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:25:41.217 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-22T08:25:41.218 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:25:41.218 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 08:20:39.280701050 +0000 2024-04-22T08:25:41.218 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 08:20:39.024703335 +0000 2024-04-22T08:25:41.218 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 08:20:39.024703335 +0000 2024-04-22T08:25:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-22T08:25:41.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T08:25:41.271 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T08:25:41.271 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T08:25:41.271 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000320742 s, 1.6 MB/s 2024-04-22T08:25:41.273 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T08:25:41.322 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 08:20:39.536698762 +0000 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 08:20:39.532698798 +0000 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 08:20:39.532698798 +0000 2024-04-22T08:25:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-22T08:25:41.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T08:25:41.427 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T08:25:41.427 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T08:25:41.427 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000202823 s, 2.5 MB/s 2024-04-22T08:25:41.428 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T08:25:41.482 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 08:20:40.296691975 +0000 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 08:20:40.012694511 +0000 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 08:20:40.012694511 +0000 2024-04-22T08:25:41.529 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-22T08:25:41.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T08:25:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T08:25:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T08:25:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000197153 s, 2.6 MB/s 2024-04-22T08:25:41.586 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T08:25:41.634 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 08:20:40.780687652 +0000 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 08:20:40.504690117 +0000 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 08:20:40.504690117 +0000 2024-04-22T08:25:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-22T08:25:41.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T08:25:41.735 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T08:25:41.735 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T08:25:41.735 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000235749 s, 2.2 MB/s 2024-04-22T08:25:41.736 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T08:25:41.786 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:25:41.786 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T08:25:41.786 INFO:tasks.ceph:Generating config... 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-22T08:25:41.787 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2024-04-22T08:25:41.788 INFO:tasks.ceph:[global] ms type = async 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T08:25:41.788 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-22T08:25:41.788 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-22T08:25:41.788 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-22T08:25:41.788 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-22T08:25:41.788 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T08:25:41.788 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-22T08:25:41.789 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T08:25:41.790 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-22T08:25:41.790 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-22T08:25:41.790 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T08:25:41.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T08:25:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T08:25:41.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T08:25:41.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T08:25:42.011 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.49:6789]'), ('mon.b', '[v1:172.21.15.49:6790]'), ('mon.c', '[v1:172.21.15.49:6791]')] 2024-04-22T08:25:42.012 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.49:6789],[v1:172.21.15.49:6790],[v1:172.21.15.49:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-22T08:25:42.012 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:42.013 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T08:25:42.061 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.49:6789]' --addv b '[v1:172.21.15.49:6790]' --addv c '[v1:172.21.15.49:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid 7ee9519b-73a7-4b76-a33e-63cd80692d01 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = pacific 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 7ee9519b-73a7-4b76-a33e-63cd80692d01 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2024-04-22T08:25:42.148278+0000 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-04-22T08:25:42.148278+0000 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 16 (pacific) 2024-04-22T08:25:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 3 2024-04-22T08:25:42.146 INFO:teuthology.orchestra.run.smithi049.stdout:0: v1:172.21.15.49:6789/0 mon.a 2024-04-22T08:25:42.146 INFO:teuthology.orchestra.run.smithi049.stdout:1: v1:172.21.15.49:6790/0 mon.b 2024-04-22T08:25:42.146 INFO:teuthology.orchestra.run.smithi049.stdout:2: v1:172.21.15.49:6791/0 mon.c 2024-04-22T08:25:42.146 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-22T08:25:42.147 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T08:25:42.193 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7ee9519b-73a7-4b76-a33e-63cd80692d01... 2024-04-22T08:25:42.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2024-04-22T08:25:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2024-04-22T08:25:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = "[v1:172.21.15.49:6789],[v1:172.21.15.49:6790],[v1:172.21.15.49:6791]" 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon client directed command retry = 5 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon client hunt interval max multiple = 2 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon election default strategy = 3 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr beacon grace = 90 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: ms bind msgr2 = False 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: ms inject socket failures = 1000 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: ms type = async 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 7ee9519b-73a7-4b76-a33e-63cd80692d01 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2024-04-22T08:25:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: bdev async discard = True 2024-04-22T08:25:42.271 INFO:teuthology.orchestra.run.smithi049.stdout: bdev enable discard = True 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore allocator = bitmap 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore zero block detection = True 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 20 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 20 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify cached snaps = True 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify missing on start = True 2024-04-22T08:25:42.272 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock override recovery settings = True 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: osd op queue = debug_random 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: osd op queue cut off = debug_random 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 10 2024-04-22T08:25:42.273 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.274 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: mon scrub interval = 300 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2024-04-22T08:25:42.275 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.b] 2024-04-22T08:25:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.c] 2024-04-22T08:25:42.277 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T08:25:42.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T08:25:42.370 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T08:25:42.371 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:42.371 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T08:25:42.376 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:42.376 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T08:25:42.424 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:25:42.424 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:42.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T08:25:42.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T08:25:42.490 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:42.490 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:42.537 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T08:25:42.537 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T08:25:42.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T08:25:42.624 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T08:25:42.627 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T08:25:42.627 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T08:25:42.627 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T08:25:42.627 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T08:25:42.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T08:25:42.679 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:25:42.679 INFO:tasks.ceph:role: osd.0 2024-04-22T08:25:42.679 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:25:42.679 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:25:42.737 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:25:42.761 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T08:25:42.762 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T08:25:42.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T08:25:42.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T08:25:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:25:42.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T08:25:42.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T08:25:42.932 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:25:42.932 INFO:tasks.ceph:role: osd.1 2024-04-22T08:25:42.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:25:42.932 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:25:42.989 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:25:43.013 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T08:25:43.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T08:25:43.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T08:25:43.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T08:25:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T08:25:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:25:43.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T08:25:43.186 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:25:43.186 INFO:tasks.ceph:role: osd.2 2024-04-22T08:25:43.186 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:25:43.186 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:25:43.241 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:25:43.265 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T08:25:43.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T08:25:43.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T08:25:43.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T08:25:43.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: /sbin/restorecon: command not found 2024-04-22T08:25:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:25:43.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-22T08:25:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-22T08:25:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:25:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:25:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:25:43.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:43.493+0000 7fa4a7e447c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-22T08:25:46.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T08:25:46.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:46.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.569+0000 7f2bfda777c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-22T08:25:46.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.573+0000 7f2bfda777c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-22T08:25:46.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.573+0000 7f2bfda777c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:25:46.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.573+0000 7f2bfda777c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:25:46.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.573+0000 7f2bfda777c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:25:46.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:46.573+0000 7f2bfda777c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-22T08:25:49.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T08:25:49.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:49.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-22T08:25:49.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-22T08:25:49.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:25:49.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:25:49.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:25:49.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T08:25:49.685+0000 7f80215ec7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-22T08:25:52.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T08:25:52.705 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T08:25:52.705 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:52.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T08:25:52.760 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:52.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T08:25:52.817 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:52.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T08:25:52.872 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:25:52.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T08:25:52.928 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T08:25:52.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAGHyZmvfxjJRAAgoFv0GkqSTIvRYSwc+drPw== 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAHHyZmXqqdHRAAxp/xjysso5psuaVsU+HMeA== 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAKHyZm/AJJIhAAzvS5iRhO9xX6ysLOZZMzSQ== 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2024-04-22T08:25:52.983 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQANHyZmGXf8KBAA7Rvp29PN2qDJ/Rf4eG0Htg== 2024-04-22T08:25:52.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T08:25:53.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:25:53.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:25:53.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:25:53.333 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T08:25:53.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T08:25:53.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:25:53.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T08:25:53.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-22T08:25:53.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:25:53.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-22T08:25:53.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-22T08:25:53.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:25:53.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-22T08:25:53.807 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:25:53.851 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T08:25:53.852 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:25:53.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:25:53.897 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:25:53.897 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T08:25:53.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T08:25:53.900 INFO:tasks.ceph.mon.b:Started 2024-04-22T08:25:53.900 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:25:53.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:25:53.902 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:25:53.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph config set mgr mgr_pool false --force 2024-04-22T08:26:05.205 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T08:26:05.206 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T08:26:05.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T08:26:05.208 INFO:tasks.ceph.mgr.x:Started 2024-04-22T08:26:05.208 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T08:26:05.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T08:26:05.405 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-04-22T08:26:05.422 INFO:tasks.ceph:check_enable_crimson: False 2024-04-22T08:26:05.422 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T08:26:05.423 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:26:05.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T08:26:05.433 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:26:05.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T08:26:05.486 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T08:26:05.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T08:26:05.516 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:05.517+0000 7fa6ef6a8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T08:26:05.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 318c137f-1dc4-4513-bc1a-acaadc0ab20c 0 2024-04-22T08:26:05.773 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T08:26:05.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new f2692c19-8d2d-4d9d-8de5-d0d885c8f379 1 2024-04-22T08:26:05.994 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-04-22T08:26:06.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5 2 2024-04-22T08:26:06.055 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:06.053+0000 7fa6ef6a8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T08:26:06.234 INFO:teuthology.orchestra.run.smithi049.stdout:2 2024-04-22T08:26:06.253 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T08:26:06.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-22T08:26:06.281 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:06.281+0000 7fa6ef6a8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T08:26:06.282 INFO:tasks.ceph.osd.0:Started 2024-04-22T08:26:06.283 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T08:26:06.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-22T08:26:06.284 INFO:tasks.ceph.osd.1:Started 2024-04-22T08:26:06.285 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T08:26:06.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-22T08:26:06.287 INFO:tasks.ceph.osd.2:Started 2024-04-22T08:26:06.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:06.484 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:06.485 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":5,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:06.235124+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:06.501 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T08:26:06.501 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T08:26:06.803 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:06.901 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:06.901+0000 7fa6ef6a8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T08:26:06.924 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:06.925+0000 7fadd5cb27c0 -1 Falling back to public interface 2024-04-22T08:26:06.925 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:26:06.925+0000 7f73a81b97c0 -1 Falling back to public interface 2024-04-22T08:26:06.952 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:26:06.953+0000 7f03620327c0 -1 Falling back to public interface 2024-04-22T08:26:06.959 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:06.957+0000 7fa6ef6a8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T08:26:06.985 INFO:teuthology.misc.health.smithi049.stdout: 2024-04-22T08:26:06.986 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":5,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:06.235124+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:07.000 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-22T08:26:07.015 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.013+0000 7fa6ef6a8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T08:26:07.095 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.093+0000 7fa6ef6a8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T08:26:07.150 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.149+0000 7fa6ef6a8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T08:26:07.230 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.229+0000 7fa6ef6a8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T08:26:07.283 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.285+0000 7fa6ef6a8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T08:26:07.440 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.437+0000 7fa6ef6a8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T08:26:07.501 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:07.501+0000 7fa6ef6a8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T08:26:08.128 INFO:tasks.ceph.mgr.x.smithi049.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-22T08:26:08.128 INFO:tasks.ceph.mgr.x.smithi049.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-22T08:26:08.129 INFO:tasks.ceph.mgr.x.smithi049.stderr: from numpy import show_config as show_numpy_config 2024-04-22T08:26:08.208 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:08.209+0000 7fa6ef6a8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T08:26:08.358 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:08.357+0000 7fa6ef6a8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T08:26:08.524 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:08.525+0000 7fa6ef6a8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T08:26:08.590 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:08.589+0000 7fa6ef6a8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T08:26:08.730 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:08.729+0000 7fa6ef6a8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T08:26:09.352 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:09.353+0000 7fa6ef6a8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T08:26:09.432 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:09.433+0000 7fa6ef6a8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T08:26:09.622 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:09.621+0000 7fa6ef6a8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T08:26:09.699 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:09.697+0000 7fa6ef6a8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T08:26:10.018 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:10.017+0000 7fa6ef6a8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T08:26:10.083 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:10.081+0000 7fa6ef6a8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T08:26:10.531 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:10.529+0000 7fadd5cb27c0 -1 osd.0 0 log_to_monitors true 2024-04-22T08:26:10.597 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:26:10.597+0000 7f73a81b97c0 -1 osd.2 0 log_to_monitors true 2024-04-22T08:26:10.919 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:26:10.917+0000 7f03620327c0 -1 osd.1 0 log_to_monitors true 2024-04-22T08:26:11.125 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:11.125+0000 7fa696b61640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-22T08:26:11.125 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T08:26:11.125+0000 7fa696b61640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-22T08:26:12.182 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:26:12.181+0000 7f035d7b4640 -1 osd.1 0 waiting for initial osdmap 2024-04-22T08:26:12.198 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:26:12.197+0000 7f0359e01640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:26:13.103 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:13.101+0000 7fadd1434640 -1 osd.0 0 waiting for initial osdmap 2024-04-22T08:26:13.103 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:26:13.101+0000 7f73a393b640 -1 osd.2 0 waiting for initial osdmap 2024-04-22T08:26:13.105 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:26:13.105+0000 7f739bf80640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:26:13.108 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:13.109+0000 7fadc9a79640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:26:13.303 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:13.603 INFO:teuthology.misc.health.smithi049.stdout: 2024-04-22T08:26:13.603 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":8,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:13.185711+0000","last_up_change":"2024-04-22T08:26:13.185711+0000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":23920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":23920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":23920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":23920}]},"public_addr":"172.21.15.49:6800/23920","cluster_addr":"172.21.15.49:6802/23920","heartbeat_back_addr":"172.21.15.49:6805/23920","heartbeat_front_addr":"172.21.15.49:6804/23920","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:13.620 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-22T08:26:13.620 INFO:tasks.ceph:Creating RBD pool 2024-04-22T08:26:13.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T08:26:14.211 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2024-04-22T08:26:14.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T08:26:15.218 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T08:26:15.245 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T08:26:15.246 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T08:26:15.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T08:26:15.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T08:26:15.247 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T08:26:15.576 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:15.593 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6812","nonce":23782}]},"active_addr":"172.21.15.49:6812/23782","active_change":"2024-04-22T08:26:10.108999+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1297353206}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":190173769}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1345162898}]}]} 2024-04-22T08:26:15.595 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T08:26:15.595 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T08:26:15.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:15.885 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:15.885 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:15.204981+0000","last_up_change":"2024-04-22T08:26:13.185711+0000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":23920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":23920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":23920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":23920}]},"public_addr":"172.21.15.49:6800/23920","cluster_addr":"172.21.15.49:6802/23920","heartbeat_back_addr":"172.21.15.49:6805/23920","heartbeat_front_addr":"172.21.15.49:6804/23920","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:15.903 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T08:26:15.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:16.195 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:16.195 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:15.204981+0000","last_up_change":"2024-04-22T08:26:13.185711+0000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":23920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":23920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":23920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":23920}]},"public_addr":"172.21.15.49:6800/23920","cluster_addr":"172.21.15.49:6802/23920","heartbeat_back_addr":"172.21.15.49:6805/23920","heartbeat_front_addr":"172.21.15.49:6804/23920","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:16.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:26:16.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T08:26:16.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:26:16.371 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:16.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:16.372 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:16.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:16.378 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:16.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:16.671 INFO:teuthology.orchestra.run.smithi049.stdout:34359738369 2024-04-22T08:26:16.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-04-22T08:26:16.710 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:16.722 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:16.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2024-04-22T08:26:16.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2024-04-22T08:26:17.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:17.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:17.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:17.999 INFO:teuthology.orchestra.run.smithi049.stdout:34359738369 2024-04-22T08:26:18.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-04-22T08:26:18.020 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:18.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2024-04-22T08:26:18.065 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:18.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2024-04-22T08:26:19.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:19.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:19.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:19.342 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2024-04-22T08:26:19.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2024-04-22T08:26:19.362 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:19.391 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:19.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2024-04-22T08:26:19.408 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:19.416 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:19.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2024-04-22T08:26:19.434 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:19.434 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T08:26:19.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:19.748 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:19.764 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-22T08:26:18.115492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89532,"kb_used_data":260,"kb_used_omap":4,"kb_used_meta":89211,"kb_avail":283025988,"statfs":{"total":289910292480,"available":289818611712,"internally_reserved":0,"allocated":266240,"data_stored":57386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4780,"internal_metadata":91352404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228716+0000","last_change":"2024-04-22T08:26:15.226400+0000","last_active":"2024-04-22T08:26:16.228716+0000","last_peered":"2024-04-22T08:26:16.228716+0000","last_clean":"2024-04-22T08:26:16.228716+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:16.228716+0000","last_undegraded":"2024-04-22T08:26:16.228716+0000","last_fullsized":"2024-04-22T08:26:16.228716+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:08.593151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228564+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:16.228564+0000","last_peered":"2024-04-22T08:26:16.228564+0000","last_clean":"2024-04-22T08:26:16.228564+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:16.228564+0000","last_undegraded":"2024-04-22T08:26:16.228564+0000","last_fullsized":"2024-04-22T08:26:16.228564+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.220145+0000","last_change":"2024-04-22T08:26:15.217054+0000","last_active":"2024-04-22T08:26:15.220145+0000","last_peered":"2024-04-22T08:26:15.220145+0000","last_clean":"2024-04-22T08:26:15.220145+0000","last_became_active":"2024-04-22T08:26:15.216307+0000","last_became_peered":"2024-04-22T08:26:15.216307+0000","last_unstale":"2024-04-22T08:26:15.220145+0000","last_undegraded":"2024-04-22T08:26:15.220145+0000","last_fullsized":"2024-04-22T08:26:15.220145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:51.423204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228698+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:16.228698+0000","last_peered":"2024-04-22T08:26:16.228698+0000","last_clean":"2024-04-22T08:26:16.228698+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:16.228698+0000","last_undegraded":"2024-04-22T08:26:16.228698+0000","last_fullsized":"2024-04-22T08:26:16.228698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228609+0000","last_change":"2024-04-22T08:26:15.226855+0000","last_active":"2024-04-22T08:26:16.228609+0000","last_peered":"2024-04-22T08:26:16.228609+0000","last_clean":"2024-04-22T08:26:16.228609+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:16.228609+0000","last_undegraded":"2024-04-22T08:26:16.228609+0000","last_fullsized":"2024-04-22T08:26:16.228609+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:17.322725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.228676+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:15.228676+0000","last_peered":"2024-04-22T08:26:15.228676+0000","last_clean":"2024-04-22T08:26:15.228676+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:15.228676+0000","last_undegraded":"2024-04-22T08:26:15.228676+0000","last_fullsized":"2024-04-22T08:26:15.228676+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.221256+0000","last_change":"2024-04-22T08:26:15.216982+0000","last_active":"2024-04-22T08:26:15.221256+0000","last_peered":"2024-04-22T08:26:15.221256+0000","last_clean":"2024-04-22T08:26:15.221256+0000","last_became_active":"2024-04-22T08:26:15.216171+0000","last_became_peered":"2024-04-22T08:26:15.216171+0000","last_unstale":"2024-04-22T08:26:15.221256+0000","last_undegraded":"2024-04-22T08:26:15.221256+0000","last_fullsized":"2024-04-22T08:26:15.221256+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:49.115592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228492+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:16.228492+0000","last_peered":"2024-04-22T08:26:16.228492+0000","last_clean":"2024-04-22T08:26:16.228492+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:16.228492+0000","last_undegraded":"2024-04-22T08:26:16.228492+0000","last_fullsized":"2024-04-22T08:26:16.228492+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31212,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340628,"statfs":{"total":96636764160,"available":96604803072,"internally_reserved":0,"allocated":90112,"data_stored":19374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":19374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31208,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340632,"statfs":{"total":96636764160,"available":96604807168,"internally_reserved":0,"allocated":86016,"data_stored":18638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:19.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:20.890 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:20.910 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-22T08:26:20.115864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89544,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":89211,"kb_avail":283025976,"statfs":{"total":289910292480,"available":289818599424,"internally_reserved":0,"allocated":282624,"data_stored":62730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4780,"internal_metadata":91352404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228716+0000","last_change":"2024-04-22T08:26:15.226400+0000","last_active":"2024-04-22T08:26:16.228716+0000","last_peered":"2024-04-22T08:26:16.228716+0000","last_clean":"2024-04-22T08:26:16.228716+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:16.228716+0000","last_undegraded":"2024-04-22T08:26:16.228716+0000","last_fullsized":"2024-04-22T08:26:16.228716+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:08.593151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228564+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:16.228564+0000","last_peered":"2024-04-22T08:26:16.228564+0000","last_clean":"2024-04-22T08:26:16.228564+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:16.228564+0000","last_undegraded":"2024-04-22T08:26:16.228564+0000","last_fullsized":"2024-04-22T08:26:16.228564+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.220145+0000","last_change":"2024-04-22T08:26:15.217054+0000","last_active":"2024-04-22T08:26:15.220145+0000","last_peered":"2024-04-22T08:26:15.220145+0000","last_clean":"2024-04-22T08:26:15.220145+0000","last_became_active":"2024-04-22T08:26:15.216307+0000","last_became_peered":"2024-04-22T08:26:15.216307+0000","last_unstale":"2024-04-22T08:26:15.220145+0000","last_undegraded":"2024-04-22T08:26:15.220145+0000","last_fullsized":"2024-04-22T08:26:15.220145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:51.423204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228698+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:16.228698+0000","last_peered":"2024-04-22T08:26:16.228698+0000","last_clean":"2024-04-22T08:26:16.228698+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:16.228698+0000","last_undegraded":"2024-04-22T08:26:16.228698+0000","last_fullsized":"2024-04-22T08:26:16.228698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228609+0000","last_change":"2024-04-22T08:26:15.226855+0000","last_active":"2024-04-22T08:26:16.228609+0000","last_peered":"2024-04-22T08:26:16.228609+0000","last_clean":"2024-04-22T08:26:16.228609+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:16.228609+0000","last_undegraded":"2024-04-22T08:26:16.228609+0000","last_fullsized":"2024-04-22T08:26:16.228609+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:17.322725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.355080+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:16.355080+0000","last_peered":"2024-04-22T08:26:16.355080+0000","last_clean":"2024-04-22T08:26:16.355080+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:16.355080+0000","last_undegraded":"2024-04-22T08:26:16.355080+0000","last_fullsized":"2024-04-22T08:26:16.355080+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.221256+0000","last_change":"2024-04-22T08:26:15.216982+0000","last_active":"2024-04-22T08:26:15.221256+0000","last_peered":"2024-04-22T08:26:15.221256+0000","last_clean":"2024-04-22T08:26:15.221256+0000","last_became_active":"2024-04-22T08:26:15.216171+0000","last_became_peered":"2024-04-22T08:26:15.216171+0000","last_unstale":"2024-04-22T08:26:15.221256+0000","last_undegraded":"2024-04-22T08:26:15.221256+0000","last_fullsized":"2024-04-22T08:26:15.221256+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:49.115592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228492+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:16.228492+0000","last_peered":"2024-04-22T08:26:16.228492+0000","last_clean":"2024-04-22T08:26:16.228492+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:16.228492+0000","last_undegraded":"2024-04-22T08:26:16.228492+0000","last_fullsized":"2024-04-22T08:26:16.228492+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31212,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340628,"statfs":{"total":96636764160,"available":96604803072,"internally_reserved":0,"allocated":90112,"data_stored":19374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":98304,"data_stored":22046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31208,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340632,"statfs":{"total":96636764160,"available":96604807168,"internally_reserved":0,"allocated":94208,"data_stored":21310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:20.911 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T08:26:20.911 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T08:26:20.911 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T08:26:20.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T08:26:21.245 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:21.245 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T08:26:21.264 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T08:26:21.264 INFO:teuthology.run_tasks:Running task rep_lost_unfound_delete... 2024-04-22T08:26:21.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:21.581 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:16.219176+0000","last_up_change":"2024-04-22T08:26:13.185711+0000","last_in_change":"2024-04-22T08:26:06.235124+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":23920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":23920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":23920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":23920}]},"public_addr":"172.21.15.49:6800/23920","cluster_addr":"172.21.15.49:6802/23920","heartbeat_back_addr":"172.21.15.49:6805/23920","heartbeat_front_addr":"172.21.15.49:6804/23920","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:21.598 INFO:tasks.rep_lost_unfound_delete.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-22T08:26:13.951550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T08:26:21.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-22T08:26:21.891 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 8 2024-04-22T08:26:21.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 11 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 7ee9519b-73a7-4b76-a33e-63cd80692d01 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-04-22T08:26:05.080988+0000 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-04-22T08:26:16.219176+0000 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:crush_version 4 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:full_ratio 0.95 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:backfillfull_ratio 0.9 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:nearfull_ratio 0.85 2024-04-22T08:26:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:require_min_compat_client luminous 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:min_compat_client jewel 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:require_osd_release reef 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:stretch_mode_enabled false 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 application rbd read_balance_score 1.88 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:max_osd 3 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6800/23920 v1:172.21.15.49:6802/23920 exists,up 318c137f-1dc4-4513-bc1a-acaadc0ab20c 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6808/23922 v1:172.21.15.49:6809/23922 exists,up f2692c19-8d2d-4d9d-8de5-d0d885c8f379 2024-04-22T08:26:22.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6801/23921 v1:172.21.15.49:6803/23921 exists,up 2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5 2024-04-22T08:26:22.200 INFO:tasks.rep_lost_unfound_delete.ceph_manager:['osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6800/23920 v1:172.21.15.49:6802/23920 exists,up 318c137f-1dc4-4513-bc1a-acaadc0ab20c', 'osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6808/23922 v1:172.21.15.49:6809/23922 exists,up f2692c19-8d2d-4d9d-8de5-d0d885c8f379', 'osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.49:6801/23921 v1:172.21.15.49:6803/23921 exists,up 2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5'] 2024-04-22T08:26:22.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:26:22.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T08:26:22.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:26:22.352 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T08:26:22.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:22.353 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-04-22T08:26:22.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:22.356 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T08:26:22.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:22.627 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-04-22T08:26:22.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.1 2024-04-22T08:26:22.687 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-04-22T08:26:22.688 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:22.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.2 2024-04-22T08:26:22.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738372 got 34359738371 for osd.0 2024-04-22T08:26:23.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:23.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:23.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:23.951 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-04-22T08:26:23.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.1 2024-04-22T08:26:23.989 INFO:teuthology.orchestra.run.smithi049.stdout:34359738371 2024-04-22T08:26:24.000 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-04-22T08:26:24.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738372 got 34359738371 for osd.0 2024-04-22T08:26:24.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.2 2024-04-22T08:26:24.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:25.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:25.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:26:25.283 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T08:26:25.300 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.1 2024-04-22T08:26:25.300 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:25.319 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T08:26:25.335 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738372 got 34359738373 for osd.0 2024-04-22T08:26:25.336 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:25.336 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T08:26:25.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.2 2024-04-22T08:26:25.352 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:25.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-04-22T08:26:25.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:25.680 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:25.680 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:25.697 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-22T08:26:24.116926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89544,"kb_used_data":284,"kb_used_omap":4,"kb_used_meta":89211,"kb_avail":283025976,"statfs":{"total":289910292480,"available":289818599424,"internally_reserved":0,"allocated":290816,"data_stored":65402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4780,"internal_metadata":91352404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228716+0000","last_change":"2024-04-22T08:26:15.226400+0000","last_active":"2024-04-22T08:26:16.228716+0000","last_peered":"2024-04-22T08:26:16.228716+0000","last_clean":"2024-04-22T08:26:16.228716+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:16.228716+0000","last_undegraded":"2024-04-22T08:26:16.228716+0000","last_fullsized":"2024-04-22T08:26:16.228716+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:08.593151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228564+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:16.228564+0000","last_peered":"2024-04-22T08:26:16.228564+0000","last_clean":"2024-04-22T08:26:16.228564+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:16.228564+0000","last_undegraded":"2024-04-22T08:26:16.228564+0000","last_fullsized":"2024-04-22T08:26:16.228564+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.220145+0000","last_change":"2024-04-22T08:26:15.217054+0000","last_active":"2024-04-22T08:26:15.220145+0000","last_peered":"2024-04-22T08:26:15.220145+0000","last_clean":"2024-04-22T08:26:15.220145+0000","last_became_active":"2024-04-22T08:26:15.216307+0000","last_became_peered":"2024-04-22T08:26:15.216307+0000","last_unstale":"2024-04-22T08:26:15.220145+0000","last_undegraded":"2024-04-22T08:26:15.220145+0000","last_fullsized":"2024-04-22T08:26:15.220145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:51.423204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228698+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:16.228698+0000","last_peered":"2024-04-22T08:26:16.228698+0000","last_clean":"2024-04-22T08:26:16.228698+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:16.228698+0000","last_undegraded":"2024-04-22T08:26:16.228698+0000","last_fullsized":"2024-04-22T08:26:16.228698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228609+0000","last_change":"2024-04-22T08:26:15.226855+0000","last_active":"2024-04-22T08:26:16.228609+0000","last_peered":"2024-04-22T08:26:16.228609+0000","last_clean":"2024-04-22T08:26:16.228609+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:16.228609+0000","last_undegraded":"2024-04-22T08:26:16.228609+0000","last_fullsized":"2024-04-22T08:26:16.228609+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:17.322725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.355080+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:16.355080+0000","last_peered":"2024-04-22T08:26:16.355080+0000","last_clean":"2024-04-22T08:26:16.355080+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:16.355080+0000","last_undegraded":"2024-04-22T08:26:16.355080+0000","last_fullsized":"2024-04-22T08:26:16.355080+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.221256+0000","last_change":"2024-04-22T08:26:15.216982+0000","last_active":"2024-04-22T08:26:15.221256+0000","last_peered":"2024-04-22T08:26:15.221256+0000","last_clean":"2024-04-22T08:26:15.221256+0000","last_became_active":"2024-04-22T08:26:15.216171+0000","last_became_peered":"2024-04-22T08:26:15.216171+0000","last_unstale":"2024-04-22T08:26:15.221256+0000","last_undegraded":"2024-04-22T08:26:15.221256+0000","last_fullsized":"2024-04-22T08:26:15.221256+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:49.115592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228492+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:16.228492+0000","last_peered":"2024-04-22T08:26:16.228492+0000","last_clean":"2024-04-22T08:26:16.228492+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:16.228492+0000","last_undegraded":"2024-04-22T08:26:16.228492+0000","last_fullsized":"2024-04-22T08:26:16.228492+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31212,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340628,"statfs":{"total":96636764160,"available":96604803072,"internally_reserved":0,"allocated":98304,"data_stored":22046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":98304,"data_stored":22046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31208,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340632,"statfs":{"total":96636764160,"available":96604807168,"internally_reserved":0,"allocated":94208,"data_stored":21310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:25.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:25.980 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:25.996 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-22T08:26:24.116926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89544,"kb_used_data":284,"kb_used_omap":4,"kb_used_meta":89211,"kb_avail":283025976,"statfs":{"total":289910292480,"available":289818599424,"internally_reserved":0,"allocated":290816,"data_stored":65402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4780,"internal_metadata":91352404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228716+0000","last_change":"2024-04-22T08:26:15.226400+0000","last_active":"2024-04-22T08:26:16.228716+0000","last_peered":"2024-04-22T08:26:16.228716+0000","last_clean":"2024-04-22T08:26:16.228716+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:16.228716+0000","last_undegraded":"2024-04-22T08:26:16.228716+0000","last_fullsized":"2024-04-22T08:26:16.228716+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:08.593151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228564+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:16.228564+0000","last_peered":"2024-04-22T08:26:16.228564+0000","last_clean":"2024-04-22T08:26:16.228564+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:16.228564+0000","last_undegraded":"2024-04-22T08:26:16.228564+0000","last_fullsized":"2024-04-22T08:26:16.228564+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.220145+0000","last_change":"2024-04-22T08:26:15.217054+0000","last_active":"2024-04-22T08:26:15.220145+0000","last_peered":"2024-04-22T08:26:15.220145+0000","last_clean":"2024-04-22T08:26:15.220145+0000","last_became_active":"2024-04-22T08:26:15.216307+0000","last_became_peered":"2024-04-22T08:26:15.216307+0000","last_unstale":"2024-04-22T08:26:15.220145+0000","last_undegraded":"2024-04-22T08:26:15.220145+0000","last_fullsized":"2024-04-22T08:26:15.220145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:51.423204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228698+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:16.228698+0000","last_peered":"2024-04-22T08:26:16.228698+0000","last_clean":"2024-04-22T08:26:16.228698+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:16.228698+0000","last_undegraded":"2024-04-22T08:26:16.228698+0000","last_fullsized":"2024-04-22T08:26:16.228698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228609+0000","last_change":"2024-04-22T08:26:15.226855+0000","last_active":"2024-04-22T08:26:16.228609+0000","last_peered":"2024-04-22T08:26:16.228609+0000","last_clean":"2024-04-22T08:26:16.228609+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:16.228609+0000","last_undegraded":"2024-04-22T08:26:16.228609+0000","last_fullsized":"2024-04-22T08:26:16.228609+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:33:17.322725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.355080+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:16.355080+0000","last_peered":"2024-04-22T08:26:16.355080+0000","last_clean":"2024-04-22T08:26:16.355080+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:16.355080+0000","last_undegraded":"2024-04-22T08:26:16.355080+0000","last_fullsized":"2024-04-22T08:26:16.355080+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-22T08:26:15.221256+0000","last_change":"2024-04-22T08:26:15.216982+0000","last_active":"2024-04-22T08:26:15.221256+0000","last_peered":"2024-04-22T08:26:15.221256+0000","last_clean":"2024-04-22T08:26:15.221256+0000","last_became_active":"2024-04-22T08:26:15.216171+0000","last_became_peered":"2024-04-22T08:26:15.216171+0000","last_unstale":"2024-04-22T08:26:15.221256+0000","last_undegraded":"2024-04-22T08:26:15.221256+0000","last_fullsized":"2024-04-22T08:26:15.221256+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:49.115592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T08:26:16.228492+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:16.228492+0000","last_peered":"2024-04-22T08:26:16.228492+0000","last_clean":"2024-04-22T08:26:16.228492+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:16.228492+0000","last_undegraded":"2024-04-22T08:26:16.228492+0000","last_fullsized":"2024-04-22T08:26:16.228492+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31212,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340628,"statfs":{"total":96636764160,"available":96604803072,"internally_reserved":0,"allocated":98304,"data_stored":22046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":98304,"data_stored":22046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31208,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340632,"statfs":{"total":96636764160,"available":96604807168,"internally_reserved":0,"allocated":94208,"data_stored":21310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:25.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-04-22T08:26:25.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:creating pool_name unfounddel_pool 2024-04-22T08:26:25.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unfounddel_pool 16 2024-04-22T08:26:27.282 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unfounddel_pool' created 2024-04-22T08:26:27.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unfounddel_pool rados --yes-i-really-mean-it || true 2024-04-22T08:26:28.291 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unfounddel_pool' 2024-04-22T08:26:29.325 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T08:26:29.325 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:26:29.393 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T08:26:29.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2024-04-22T08:26:29.703 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:29.704+0000 7f403222d640 -1 mon.c@2(leader).osd e14 definitely_dead 0 2024-04-22T08:26:30.306 INFO:teuthology.orchestra.run.smithi049.stderr:marked down osd.2. 2024-04-22T08:26:30.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-04-22T08:26:30.630 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:30.628+0000 7f403222d640 -1 mon.c@2(leader).osd e15 definitely_dead 0 2024-04-22T08:26:31.514 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.2. 2024-04-22T08:26:31.540 INFO:tasks.util.rados:rados -p unfounddel_pool put dummy /etc/fstab 2024-04-22T08:26:31.540 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put dummy /etc/fstab 2024-04-22T08:26:31.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:26:31.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T08:26:31.787 INFO:teuthology.orchestra.run.smithi049.stdout:34359738375 2024-04-22T08:26:31.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:31.789 INFO:teuthology.orchestra.run.smithi049.stdout:34359738376 2024-04-22T08:26:31.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:32.133 INFO:teuthology.orchestra.run.smithi049.stdout:34359738375 2024-04-22T08:26:32.136 INFO:teuthology.orchestra.run.smithi049.stdout:34359738374 2024-04-22T08:26:32.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738375 for osd.1 2024-04-22T08:26:32.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738375 got 34359738374 for osd.0 2024-04-22T08:26:33.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:33.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:33.456 INFO:teuthology.orchestra.run.smithi049.stdout:34359738376 2024-04-22T08:26:33.461 INFO:teuthology.orchestra.run.smithi049.stdout:34359738375 2024-04-22T08:26:33.483 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738376 for osd.1 2024-04-22T08:26:33.483 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:33.487 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738375 got 34359738375 for osd.0 2024-04-22T08:26:33.488 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:33.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-04-22T08:26:33.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:33.822 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:33.822 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:33.839 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-22T08:26:32.520720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58384,"kb_used_data":236,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685296,"statfs":{"total":193273528320,"available":193213743104,"internally_reserved":0,"allocated":241664,"data_stored":61662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3186,"internal_metadata":59503502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.401780"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.309167+0000","last_change":"2024-04-22T08:26:28.305412+0000","last_active":"2024-04-22T08:26:30.309167+0000","last_peered":"2024-04-22T08:26:30.309167+0000","last_clean":"2024-04-22T08:26:30.309167+0000","last_became_active":"2024-04-22T08:26:28.305105+0000","last_became_peered":"2024-04-22T08:26:28.305105+0000","last_unstale":"2024-04-22T08:26:30.309167+0000","last_undegraded":"2024-04-22T08:26:30.309167+0000","last_fullsized":"2024-04-22T08:26:30.309167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:54.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.525805+0000","last_change":"2024-04-22T08:26:31.525805+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.525805+0000","last_undegraded":"2024-04-22T08:26:31.525805+0000","last_fullsized":"2024-04-22T08:26:31.525805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:22:00.429929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518322+0000","last_change":"2024-04-22T08:26:31.518641+0000","last_active":"2024-04-22T08:26:30.805842+0000","last_peered":"2024-04-22T08:26:29.186781+0000","last_clean":"2024-04-22T08:26:29.186781+0000","last_became_active":"2024-04-22T08:26:28.304346+0000","last_became_peered":"2024-04-22T08:26:28.304346+0000","last_unstale":"2024-04-22T08:26:31.518322+0000","last_undegraded":"2024-04-22T08:26:31.518322+0000","last_fullsized":"2024-04-22T08:26:31.518322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:47.072408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.524958+0000","last_change":"2024-04-22T08:26:31.524958+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.524958+0000","last_undegraded":"2024-04-22T08:26:31.524958+0000","last_fullsized":"2024-04-22T08:26:31.524958+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:27:07.073549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.526319+0000","last_change":"2024-04-22T08:26:31.526451+0000","last_active":"2024-04-22T08:26:30.306928+0000","last_peered":"2024-04-22T08:26:29.290887+0000","last_clean":"2024-04-22T08:26:29.290887+0000","last_became_active":"2024-04-22T08:26:28.298237+0000","last_became_peered":"2024-04-22T08:26:28.298237+0000","last_unstale":"2024-04-22T08:26:31.526319+0000","last_undegraded":"2024-04-22T08:26:31.526319+0000","last_fullsized":"2024-04-22T08:26:31.526319+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:17.842260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803306+0000","last_change":"2024-04-22T08:26:28.308830+0000","last_active":"2024-04-22T08:26:30.803306+0000","last_peered":"2024-04-22T08:26:30.803306+0000","last_clean":"2024-04-22T08:26:30.803306+0000","last_became_active":"2024-04-22T08:26:28.308404+0000","last_became_peered":"2024-04-22T08:26:28.308404+0000","last_unstale":"2024-04-22T08:26:30.803306+0000","last_undegraded":"2024-04-22T08:26:30.803306+0000","last_fullsized":"2024-04-22T08:26:30.803306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:50.451518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804158+0000","last_change":"2024-04-22T08:26:28.307143+0000","last_active":"2024-04-22T08:26:30.804158+0000","last_peered":"2024-04-22T08:26:30.804158+0000","last_clean":"2024-04-22T08:26:30.804158+0000","last_became_active":"2024-04-22T08:26:28.306710+0000","last_became_peered":"2024-04-22T08:26:28.306710+0000","last_unstale":"2024-04-22T08:26:30.804158+0000","last_undegraded":"2024-04-22T08:26:30.804158+0000","last_fullsized":"2024-04-22T08:26:30.804158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:01.205072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.523822+0000","last_change":"2024-04-22T08:26:31.524160+0000","last_active":"2024-04-22T08:26:30.308975+0000","last_peered":"2024-04-22T08:26:29.291065+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:26:28.297657+0000","last_became_peered":"2024-04-22T08:26:28.297657+0000","last_unstale":"2024-04-22T08:26:31.523822+0000","last_undegraded":"2024-04-22T08:26:31.523822+0000","last_fullsized":"2024-04-22T08:26:31.523822+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:12:38.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803269+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:30.803269+0000","last_peered":"2024-04-22T08:26:30.803269+0000","last_clean":"2024-04-22T08:26:30.803269+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:30.803269+0000","last_undegraded":"2024-04-22T08:26:30.803269+0000","last_fullsized":"2024-04-22T08:26:30.803269+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.521560+0000","last_change":"2024-04-22T08:26:31.521779+0000","last_active":"2024-04-22T08:26:30.804300+0000","last_peered":"2024-04-22T08:26:29.186660+0000","last_clean":"2024-04-22T08:26:29.186660+0000","last_became_active":"2024-04-22T08:26:28.304506+0000","last_became_peered":"2024-04-22T08:26:28.304506+0000","last_unstale":"2024-04-22T08:26:31.521560+0000","last_undegraded":"2024-04-22T08:26:31.521560+0000","last_fullsized":"2024-04-22T08:26:31.521560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:59.567376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":6,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.524123+0000","last_change":"2024-04-22T08:26:31.524123+0000","last_active":"2024-04-22T08:26:14.194929+0000","last_peered":"2024-04-22T08:26:14.194929+0000","last_clean":"2024-04-22T08:26:14.194929+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.524123+0000","last_undegraded":"2024-04-22T08:26:31.524123+0000","last_fullsized":"2024-04-22T08:26:31.524123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:12.320178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.305740+0000","last_change":"2024-04-22T08:26:28.305575+0000","last_active":"2024-04-22T08:26:30.305740+0000","last_peered":"2024-04-22T08:26:30.305740+0000","last_clean":"2024-04-22T08:26:30.305740+0000","last_became_active":"2024-04-22T08:26:28.305202+0000","last_became_peered":"2024-04-22T08:26:28.305202+0000","last_unstale":"2024-04-22T08:26:30.305740+0000","last_undegraded":"2024-04-22T08:26:30.305740+0000","last_fullsized":"2024-04-22T08:26:30.305740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:12.327630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.305992+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:30.305992+0000","last_peered":"2024-04-22T08:26:30.305992+0000","last_clean":"2024-04-22T08:26:30.305992+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:30.305992+0000","last_undegraded":"2024-04-22T08:26:30.305992+0000","last_fullsized":"2024-04-22T08:26:30.305992+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518883+0000","last_change":"2024-04-22T08:26:31.518883+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518883+0000","last_undegraded":"2024-04-22T08:26:31.518883+0000","last_fullsized":"2024-04-22T08:26:31.518883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:05:30.643641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.519293+0000","last_change":"2024-04-22T08:26:31.519410+0000","last_active":"2024-04-22T08:26:30.805718+0000","last_peered":"2024-04-22T08:26:28.300852+0000","last_clean":"2024-04-22T08:26:28.300852+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:31.519293+0000","last_undegraded":"2024-04-22T08:26:31.519293+0000","last_fullsized":"2024-04-22T08:26:31.519293+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:14.170849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518001+0000","last_change":"2024-04-22T08:26:31.518001+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518001+0000","last_undegraded":"2024-04-22T08:26:31.518001+0000","last_fullsized":"2024-04-22T08:26:31.518001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:54.368834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803972+0000","last_change":"2024-04-22T08:26:28.307119+0000","last_active":"2024-04-22T08:26:30.803972+0000","last_peered":"2024-04-22T08:26:30.803972+0000","last_clean":"2024-04-22T08:26:30.803972+0000","last_became_active":"2024-04-22T08:26:28.306723+0000","last_became_peered":"2024-04-22T08:26:28.306723+0000","last_unstale":"2024-04-22T08:26:30.803972+0000","last_undegraded":"2024-04-22T08:26:30.803972+0000","last_fullsized":"2024-04-22T08:26:30.803972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:08.281247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804096+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:30.804096+0000","last_peered":"2024-04-22T08:26:30.804096+0000","last_clean":"2024-04-22T08:26:30.804096+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:30.804096+0000","last_undegraded":"2024-04-22T08:26:30.804096+0000","last_fullsized":"2024-04-22T08:26:30.804096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803772+0000","last_change":"2024-04-22T08:26:28.307051+0000","last_active":"2024-04-22T08:26:30.803772+0000","last_peered":"2024-04-22T08:26:30.803772+0000","last_clean":"2024-04-22T08:26:30.803772+0000","last_became_active":"2024-04-22T08:26:28.306684+0000","last_became_peered":"2024-04-22T08:26:28.306684+0000","last_unstale":"2024-04-22T08:26:30.803772+0000","last_undegraded":"2024-04-22T08:26:30.803772+0000","last_fullsized":"2024-04-22T08:26:30.803772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:09.266110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":6,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.525332+0000","last_change":"2024-04-22T08:26:31.525332+0000","last_active":"2024-04-22T08:26:14.194929+0000","last_peered":"2024-04-22T08:26:14.194929+0000","last_clean":"2024-04-22T08:26:14.194929+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.525332+0000","last_undegraded":"2024-04-22T08:26:31.525332+0000","last_fullsized":"2024-04-22T08:26:31.525332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:03:16.883365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804095+0000","last_change":"2024-04-22T08:26:28.308779+0000","last_active":"2024-04-22T08:26:30.804095+0000","last_peered":"2024-04-22T08:26:30.804095+0000","last_clean":"2024-04-22T08:26:30.804095+0000","last_became_active":"2024-04-22T08:26:28.308412+0000","last_became_peered":"2024-04-22T08:26:28.308412+0000","last_unstale":"2024-04-22T08:26:30.804095+0000","last_undegraded":"2024-04-22T08:26:30.804095+0000","last_fullsized":"2024-04-22T08:26:30.804095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:38.458799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804099+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:30.804099+0000","last_peered":"2024-04-22T08:26:30.804099+0000","last_clean":"2024-04-22T08:26:30.804099+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:30.804099+0000","last_undegraded":"2024-04-22T08:26:30.804099+0000","last_fullsized":"2024-04-22T08:26:30.804099+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518850+0000","last_change":"2024-04-22T08:26:31.519165+0000","last_active":"2024-04-22T08:26:30.806001+0000","last_peered":"2024-04-22T08:26:28.302571+0000","last_clean":"2024-04-22T08:26:28.302571+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:31.518850+0000","last_undegraded":"2024-04-22T08:26:31.518850+0000","last_fullsized":"2024-04-22T08:26:31.518850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:40.225598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"16'1","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517387+0000","last_change":"2024-04-22T08:26:28.308992+0000","last_active":"2024-04-22T08:26:31.517387+0000","last_peered":"2024-04-22T08:26:31.517387+0000","last_clean":"2024-04-22T08:26:31.517387+0000","last_became_active":"2024-04-22T08:26:28.308543+0000","last_became_peered":"2024-04-22T08:26:28.308543+0000","last_unstale":"2024-04-22T08:26:31.517387+0000","last_undegraded":"2024-04-22T08:26:31.517387+0000","last_fullsized":"2024-04-22T08:26:31.517387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:45.907262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":28173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":126976,"data_stored":33489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:33.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:34.121 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:34.121 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:34.138 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-22T08:26:32.520720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58384,"kb_used_data":236,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685296,"statfs":{"total":193273528320,"available":193213743104,"internally_reserved":0,"allocated":241664,"data_stored":61662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3186,"internal_metadata":59503502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.401780"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.309167+0000","last_change":"2024-04-22T08:26:28.305412+0000","last_active":"2024-04-22T08:26:30.309167+0000","last_peered":"2024-04-22T08:26:30.309167+0000","last_clean":"2024-04-22T08:26:30.309167+0000","last_became_active":"2024-04-22T08:26:28.305105+0000","last_became_peered":"2024-04-22T08:26:28.305105+0000","last_unstale":"2024-04-22T08:26:30.309167+0000","last_undegraded":"2024-04-22T08:26:30.309167+0000","last_fullsized":"2024-04-22T08:26:30.309167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:54.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.525805+0000","last_change":"2024-04-22T08:26:31.525805+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.525805+0000","last_undegraded":"2024-04-22T08:26:31.525805+0000","last_fullsized":"2024-04-22T08:26:31.525805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:22:00.429929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518322+0000","last_change":"2024-04-22T08:26:31.518641+0000","last_active":"2024-04-22T08:26:30.805842+0000","last_peered":"2024-04-22T08:26:29.186781+0000","last_clean":"2024-04-22T08:26:29.186781+0000","last_became_active":"2024-04-22T08:26:28.304346+0000","last_became_peered":"2024-04-22T08:26:28.304346+0000","last_unstale":"2024-04-22T08:26:31.518322+0000","last_undegraded":"2024-04-22T08:26:31.518322+0000","last_fullsized":"2024-04-22T08:26:31.518322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:47.072408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.524958+0000","last_change":"2024-04-22T08:26:31.524958+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.524958+0000","last_undegraded":"2024-04-22T08:26:31.524958+0000","last_fullsized":"2024-04-22T08:26:31.524958+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:27:07.073549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.526319+0000","last_change":"2024-04-22T08:26:31.526451+0000","last_active":"2024-04-22T08:26:30.306928+0000","last_peered":"2024-04-22T08:26:29.290887+0000","last_clean":"2024-04-22T08:26:29.290887+0000","last_became_active":"2024-04-22T08:26:28.298237+0000","last_became_peered":"2024-04-22T08:26:28.298237+0000","last_unstale":"2024-04-22T08:26:31.526319+0000","last_undegraded":"2024-04-22T08:26:31.526319+0000","last_fullsized":"2024-04-22T08:26:31.526319+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:17.842260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803306+0000","last_change":"2024-04-22T08:26:28.308830+0000","last_active":"2024-04-22T08:26:30.803306+0000","last_peered":"2024-04-22T08:26:30.803306+0000","last_clean":"2024-04-22T08:26:30.803306+0000","last_became_active":"2024-04-22T08:26:28.308404+0000","last_became_peered":"2024-04-22T08:26:28.308404+0000","last_unstale":"2024-04-22T08:26:30.803306+0000","last_undegraded":"2024-04-22T08:26:30.803306+0000","last_fullsized":"2024-04-22T08:26:30.803306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:50.451518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804158+0000","last_change":"2024-04-22T08:26:28.307143+0000","last_active":"2024-04-22T08:26:30.804158+0000","last_peered":"2024-04-22T08:26:30.804158+0000","last_clean":"2024-04-22T08:26:30.804158+0000","last_became_active":"2024-04-22T08:26:28.306710+0000","last_became_peered":"2024-04-22T08:26:28.306710+0000","last_unstale":"2024-04-22T08:26:30.804158+0000","last_undegraded":"2024-04-22T08:26:30.804158+0000","last_fullsized":"2024-04-22T08:26:30.804158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:01.205072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.523822+0000","last_change":"2024-04-22T08:26:31.524160+0000","last_active":"2024-04-22T08:26:30.308975+0000","last_peered":"2024-04-22T08:26:29.291065+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:26:28.297657+0000","last_became_peered":"2024-04-22T08:26:28.297657+0000","last_unstale":"2024-04-22T08:26:31.523822+0000","last_undegraded":"2024-04-22T08:26:31.523822+0000","last_fullsized":"2024-04-22T08:26:31.523822+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:12:38.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803269+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:30.803269+0000","last_peered":"2024-04-22T08:26:30.803269+0000","last_clean":"2024-04-22T08:26:30.803269+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:30.803269+0000","last_undegraded":"2024-04-22T08:26:30.803269+0000","last_fullsized":"2024-04-22T08:26:30.803269+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.521560+0000","last_change":"2024-04-22T08:26:31.521779+0000","last_active":"2024-04-22T08:26:30.804300+0000","last_peered":"2024-04-22T08:26:29.186660+0000","last_clean":"2024-04-22T08:26:29.186660+0000","last_became_active":"2024-04-22T08:26:28.304506+0000","last_became_peered":"2024-04-22T08:26:28.304506+0000","last_unstale":"2024-04-22T08:26:31.521560+0000","last_undegraded":"2024-04-22T08:26:31.521560+0000","last_fullsized":"2024-04-22T08:26:31.521560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:59.567376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":6,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.524123+0000","last_change":"2024-04-22T08:26:31.524123+0000","last_active":"2024-04-22T08:26:14.194929+0000","last_peered":"2024-04-22T08:26:14.194929+0000","last_clean":"2024-04-22T08:26:14.194929+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.524123+0000","last_undegraded":"2024-04-22T08:26:31.524123+0000","last_fullsized":"2024-04-22T08:26:31.524123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:12.320178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.305740+0000","last_change":"2024-04-22T08:26:28.305575+0000","last_active":"2024-04-22T08:26:30.305740+0000","last_peered":"2024-04-22T08:26:30.305740+0000","last_clean":"2024-04-22T08:26:30.305740+0000","last_became_active":"2024-04-22T08:26:28.305202+0000","last_became_peered":"2024-04-22T08:26:28.305202+0000","last_unstale":"2024-04-22T08:26:30.305740+0000","last_undegraded":"2024-04-22T08:26:30.305740+0000","last_fullsized":"2024-04-22T08:26:30.305740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:12.327630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.305992+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:30.305992+0000","last_peered":"2024-04-22T08:26:30.305992+0000","last_clean":"2024-04-22T08:26:30.305992+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:30.305992+0000","last_undegraded":"2024-04-22T08:26:30.305992+0000","last_fullsized":"2024-04-22T08:26:30.305992+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518883+0000","last_change":"2024-04-22T08:26:31.518883+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518883+0000","last_undegraded":"2024-04-22T08:26:31.518883+0000","last_fullsized":"2024-04-22T08:26:31.518883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:05:30.643641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.519293+0000","last_change":"2024-04-22T08:26:31.519410+0000","last_active":"2024-04-22T08:26:30.805718+0000","last_peered":"2024-04-22T08:26:28.300852+0000","last_clean":"2024-04-22T08:26:28.300852+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:31.519293+0000","last_undegraded":"2024-04-22T08:26:31.519293+0000","last_fullsized":"2024-04-22T08:26:31.519293+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:14.170849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518001+0000","last_change":"2024-04-22T08:26:31.518001+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518001+0000","last_undegraded":"2024-04-22T08:26:31.518001+0000","last_fullsized":"2024-04-22T08:26:31.518001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:54.368834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803972+0000","last_change":"2024-04-22T08:26:28.307119+0000","last_active":"2024-04-22T08:26:30.803972+0000","last_peered":"2024-04-22T08:26:30.803972+0000","last_clean":"2024-04-22T08:26:30.803972+0000","last_became_active":"2024-04-22T08:26:28.306723+0000","last_became_peered":"2024-04-22T08:26:28.306723+0000","last_unstale":"2024-04-22T08:26:30.803972+0000","last_undegraded":"2024-04-22T08:26:30.803972+0000","last_fullsized":"2024-04-22T08:26:30.803972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:08.281247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804096+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:30.804096+0000","last_peered":"2024-04-22T08:26:30.804096+0000","last_clean":"2024-04-22T08:26:30.804096+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:30.804096+0000","last_undegraded":"2024-04-22T08:26:30.804096+0000","last_fullsized":"2024-04-22T08:26:30.804096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803772+0000","last_change":"2024-04-22T08:26:28.307051+0000","last_active":"2024-04-22T08:26:30.803772+0000","last_peered":"2024-04-22T08:26:30.803772+0000","last_clean":"2024-04-22T08:26:30.803772+0000","last_became_active":"2024-04-22T08:26:28.306684+0000","last_became_peered":"2024-04-22T08:26:28.306684+0000","last_unstale":"2024-04-22T08:26:30.803772+0000","last_undegraded":"2024-04-22T08:26:30.803772+0000","last_fullsized":"2024-04-22T08:26:30.803772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:09.266110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":6,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.525332+0000","last_change":"2024-04-22T08:26:31.525332+0000","last_active":"2024-04-22T08:26:14.194929+0000","last_peered":"2024-04-22T08:26:14.194929+0000","last_clean":"2024-04-22T08:26:14.194929+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.525332+0000","last_undegraded":"2024-04-22T08:26:31.525332+0000","last_fullsized":"2024-04-22T08:26:31.525332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:03:16.883365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804095+0000","last_change":"2024-04-22T08:26:28.308779+0000","last_active":"2024-04-22T08:26:30.804095+0000","last_peered":"2024-04-22T08:26:30.804095+0000","last_clean":"2024-04-22T08:26:30.804095+0000","last_became_active":"2024-04-22T08:26:28.308412+0000","last_became_peered":"2024-04-22T08:26:28.308412+0000","last_unstale":"2024-04-22T08:26:30.804095+0000","last_undegraded":"2024-04-22T08:26:30.804095+0000","last_fullsized":"2024-04-22T08:26:30.804095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:38.458799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804099+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:30.804099+0000","last_peered":"2024-04-22T08:26:30.804099+0000","last_clean":"2024-04-22T08:26:30.804099+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:30.804099+0000","last_undegraded":"2024-04-22T08:26:30.804099+0000","last_fullsized":"2024-04-22T08:26:30.804099+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518850+0000","last_change":"2024-04-22T08:26:31.519165+0000","last_active":"2024-04-22T08:26:30.806001+0000","last_peered":"2024-04-22T08:26:28.302571+0000","last_clean":"2024-04-22T08:26:28.302571+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:31.518850+0000","last_undegraded":"2024-04-22T08:26:31.518850+0000","last_fullsized":"2024-04-22T08:26:31.518850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:40.225598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"16'1","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517387+0000","last_change":"2024-04-22T08:26:28.308992+0000","last_active":"2024-04-22T08:26:31.517387+0000","last_peered":"2024-04-22T08:26:31.517387+0000","last_clean":"2024-04-22T08:26:31.517387+0000","last_became_active":"2024-04-22T08:26:28.308543+0000","last_became_peered":"2024-04-22T08:26:28.308543+0000","last_unstale":"2024-04-22T08:26:31.517387+0000","last_undegraded":"2024-04-22T08:26:31.517387+0000","last_fullsized":"2024-04-22T08:26:31.517387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:45.907262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":28173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":126976,"data_stored":33489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:34.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:34.438 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:34.438 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:34.456 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-22T08:26:34.124087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58388,"kb_used_data":248,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685292,"statfs":{"total":193273528320,"available":193213739008,"internally_reserved":0,"allocated":253952,"data_stored":65054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3186,"internal_metadata":59503502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.005146"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.526511+0000","last_change":"2024-04-22T08:26:28.305412+0000","last_active":"2024-04-22T08:26:31.526511+0000","last_peered":"2024-04-22T08:26:31.526511+0000","last_clean":"2024-04-22T08:26:31.526511+0000","last_became_active":"2024-04-22T08:26:28.305105+0000","last_became_peered":"2024-04-22T08:26:28.305105+0000","last_unstale":"2024-04-22T08:26:31.526511+0000","last_undegraded":"2024-04-22T08:26:31.526511+0000","last_fullsized":"2024-04-22T08:26:31.526511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:54.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535217+0000","last_change":"2024-04-22T08:26:32.535217+0000","last_active":"2024-04-22T08:26:32.535217+0000","last_peered":"2024-04-22T08:26:32.535217+0000","last_clean":"2024-04-22T08:26:32.535217+0000","last_became_active":"2024-04-22T08:26:32.534389+0000","last_became_peered":"2024-04-22T08:26:32.534389+0000","last_unstale":"2024-04-22T08:26:32.535217+0000","last_undegraded":"2024-04-22T08:26:32.535217+0000","last_fullsized":"2024-04-22T08:26:32.535217+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:22:00.429929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518322+0000","last_change":"2024-04-22T08:26:31.518641+0000","last_active":"2024-04-22T08:26:30.805842+0000","last_peered":"2024-04-22T08:26:29.186781+0000","last_clean":"2024-04-22T08:26:29.186781+0000","last_became_active":"2024-04-22T08:26:28.304346+0000","last_became_peered":"2024-04-22T08:26:28.304346+0000","last_unstale":"2024-04-22T08:26:31.518322+0000","last_undegraded":"2024-04-22T08:26:31.518322+0000","last_fullsized":"2024-04-22T08:26:31.518322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:46:47.072408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535237+0000","last_change":"2024-04-22T08:26:32.535237+0000","last_active":"2024-04-22T08:26:32.535237+0000","last_peered":"2024-04-22T08:26:32.535237+0000","last_clean":"2024-04-22T08:26:32.535237+0000","last_became_active":"2024-04-22T08:26:32.534386+0000","last_became_peered":"2024-04-22T08:26:32.534386+0000","last_unstale":"2024-04-22T08:26:32.535237+0000","last_undegraded":"2024-04-22T08:26:32.535237+0000","last_fullsized":"2024-04-22T08:26:32.535237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:27:07.073549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535911+0000","last_change":"2024-04-22T08:26:32.535911+0000","last_active":"2024-04-22T08:26:32.535911+0000","last_peered":"2024-04-22T08:26:32.535911+0000","last_clean":"2024-04-22T08:26:32.535911+0000","last_became_active":"2024-04-22T08:26:32.535336+0000","last_became_peered":"2024-04-22T08:26:32.535336+0000","last_unstale":"2024-04-22T08:26:32.535911+0000","last_undegraded":"2024-04-22T08:26:32.535911+0000","last_fullsized":"2024-04-22T08:26:32.535911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:11:31.743349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803306+0000","last_change":"2024-04-22T08:26:28.308830+0000","last_active":"2024-04-22T08:26:30.803306+0000","last_peered":"2024-04-22T08:26:30.803306+0000","last_clean":"2024-04-22T08:26:30.803306+0000","last_became_active":"2024-04-22T08:26:28.308404+0000","last_became_peered":"2024-04-22T08:26:28.308404+0000","last_unstale":"2024-04-22T08:26:30.803306+0000","last_undegraded":"2024-04-22T08:26:30.803306+0000","last_fullsized":"2024-04-22T08:26:30.803306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:50.451518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804158+0000","last_change":"2024-04-22T08:26:28.307143+0000","last_active":"2024-04-22T08:26:30.804158+0000","last_peered":"2024-04-22T08:26:30.804158+0000","last_clean":"2024-04-22T08:26:30.804158+0000","last_became_active":"2024-04-22T08:26:28.306710+0000","last_became_peered":"2024-04-22T08:26:28.306710+0000","last_unstale":"2024-04-22T08:26:30.804158+0000","last_undegraded":"2024-04-22T08:26:30.804158+0000","last_fullsized":"2024-04-22T08:26:30.804158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:01.205072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535379+0000","last_change":"2024-04-22T08:26:32.535379+0000","last_active":"2024-04-22T08:26:32.535379+0000","last_peered":"2024-04-22T08:26:32.535379+0000","last_clean":"2024-04-22T08:26:32.535379+0000","last_became_active":"2024-04-22T08:26:32.534533+0000","last_became_peered":"2024-04-22T08:26:32.534533+0000","last_unstale":"2024-04-22T08:26:32.535379+0000","last_undegraded":"2024-04-22T08:26:32.535379+0000","last_fullsized":"2024-04-22T08:26:32.535379+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:50.440072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803269+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:30.803269+0000","last_peered":"2024-04-22T08:26:30.803269+0000","last_clean":"2024-04-22T08:26:30.803269+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:30.803269+0000","last_undegraded":"2024-04-22T08:26:30.803269+0000","last_fullsized":"2024-04-22T08:26:30.803269+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.521560+0000","last_change":"2024-04-22T08:26:31.521779+0000","last_active":"2024-04-22T08:26:30.804300+0000","last_peered":"2024-04-22T08:26:29.186660+0000","last_clean":"2024-04-22T08:26:29.186660+0000","last_became_active":"2024-04-22T08:26:28.304506+0000","last_became_peered":"2024-04-22T08:26:28.304506+0000","last_unstale":"2024-04-22T08:26:31.521560+0000","last_undegraded":"2024-04-22T08:26:31.521560+0000","last_fullsized":"2024-04-22T08:26:31.521560+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:55:59.567376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535075+0000","last_change":"2024-04-22T08:26:32.535075+0000","last_active":"2024-04-22T08:26:32.535075+0000","last_peered":"2024-04-22T08:26:32.535075+0000","last_clean":"2024-04-22T08:26:32.535075+0000","last_became_active":"2024-04-22T08:26:32.534036+0000","last_became_peered":"2024-04-22T08:26:32.534036+0000","last_unstale":"2024-04-22T08:26:32.535075+0000","last_undegraded":"2024-04-22T08:26:32.535075+0000","last_fullsized":"2024-04-22T08:26:32.535075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:12.320178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.523723+0000","last_change":"2024-04-22T08:26:28.305575+0000","last_active":"2024-04-22T08:26:31.523723+0000","last_peered":"2024-04-22T08:26:31.523723+0000","last_clean":"2024-04-22T08:26:31.523723+0000","last_became_active":"2024-04-22T08:26:28.305202+0000","last_became_peered":"2024-04-22T08:26:28.305202+0000","last_unstale":"2024-04-22T08:26:31.523723+0000","last_undegraded":"2024-04-22T08:26:31.523723+0000","last_fullsized":"2024-04-22T08:26:31.523723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:12.327630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.523814+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:31.523814+0000","last_peered":"2024-04-22T08:26:31.523814+0000","last_clean":"2024-04-22T08:26:31.523814+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:31.523814+0000","last_undegraded":"2024-04-22T08:26:31.523814+0000","last_fullsized":"2024-04-22T08:26:31.523814+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518883+0000","last_change":"2024-04-22T08:26:31.518883+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518883+0000","last_undegraded":"2024-04-22T08:26:31.518883+0000","last_fullsized":"2024-04-22T08:26:31.518883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:05:30.643641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.519293+0000","last_change":"2024-04-22T08:26:31.519410+0000","last_active":"2024-04-22T08:26:30.805718+0000","last_peered":"2024-04-22T08:26:28.300852+0000","last_clean":"2024-04-22T08:26:28.300852+0000","last_became_active":"2024-04-22T08:26:15.226573+0000","last_became_peered":"2024-04-22T08:26:15.226573+0000","last_unstale":"2024-04-22T08:26:31.519293+0000","last_undegraded":"2024-04-22T08:26:31.519293+0000","last_fullsized":"2024-04-22T08:26:31.519293+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:14.170849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518001+0000","last_change":"2024-04-22T08:26:31.518001+0000","last_active":"2024-04-22T08:26:27.267649+0000","last_peered":"2024-04-22T08:26:27.267649+0000","last_clean":"2024-04-22T08:26:27.267649+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T08:26:31.518001+0000","last_undegraded":"2024-04-22T08:26:31.518001+0000","last_fullsized":"2024-04-22T08:26:31.518001+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:54.368834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803972+0000","last_change":"2024-04-22T08:26:28.307119+0000","last_active":"2024-04-22T08:26:30.803972+0000","last_peered":"2024-04-22T08:26:30.803972+0000","last_clean":"2024-04-22T08:26:30.803972+0000","last_became_active":"2024-04-22T08:26:28.306723+0000","last_became_peered":"2024-04-22T08:26:28.306723+0000","last_unstale":"2024-04-22T08:26:30.803972+0000","last_undegraded":"2024-04-22T08:26:30.803972+0000","last_fullsized":"2024-04-22T08:26:30.803972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:08.281247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804096+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:30.804096+0000","last_peered":"2024-04-22T08:26:30.804096+0000","last_clean":"2024-04-22T08:26:30.804096+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:30.804096+0000","last_undegraded":"2024-04-22T08:26:30.804096+0000","last_fullsized":"2024-04-22T08:26:30.804096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.803772+0000","last_change":"2024-04-22T08:26:28.307051+0000","last_active":"2024-04-22T08:26:30.803772+0000","last_peered":"2024-04-22T08:26:30.803772+0000","last_clean":"2024-04-22T08:26:30.803772+0000","last_became_active":"2024-04-22T08:26:28.306684+0000","last_became_peered":"2024-04-22T08:26:28.306684+0000","last_unstale":"2024-04-22T08:26:30.803772+0000","last_undegraded":"2024-04-22T08:26:30.803772+0000","last_fullsized":"2024-04-22T08:26:30.803772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:09.266110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.540199+0000","last_change":"2024-04-22T08:26:32.540199+0000","last_active":"2024-04-22T08:26:32.540199+0000","last_peered":"2024-04-22T08:26:32.540199+0000","last_clean":"2024-04-22T08:26:32.540199+0000","last_became_active":"2024-04-22T08:26:32.539618+0000","last_became_peered":"2024-04-22T08:26:32.539618+0000","last_unstale":"2024-04-22T08:26:32.540199+0000","last_undegraded":"2024-04-22T08:26:32.540199+0000","last_fullsized":"2024-04-22T08:26:32.540199+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:03:16.883365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804095+0000","last_change":"2024-04-22T08:26:28.308779+0000","last_active":"2024-04-22T08:26:30.804095+0000","last_peered":"2024-04-22T08:26:30.804095+0000","last_clean":"2024-04-22T08:26:30.804095+0000","last_became_active":"2024-04-22T08:26:28.308412+0000","last_became_peered":"2024-04-22T08:26:28.308412+0000","last_unstale":"2024-04-22T08:26:30.804095+0000","last_undegraded":"2024-04-22T08:26:30.804095+0000","last_fullsized":"2024-04-22T08:26:30.804095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:38.458799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:26:30.804099+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:30.804099+0000","last_peered":"2024-04-22T08:26:30.804099+0000","last_clean":"2024-04-22T08:26:30.804099+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:30.804099+0000","last_undegraded":"2024-04-22T08:26:30.804099+0000","last_fullsized":"2024-04-22T08:26:30.804099+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"peering","last_fresh":"2024-04-22T08:26:31.518850+0000","last_change":"2024-04-22T08:26:31.519165+0000","last_active":"2024-04-22T08:26:30.806001+0000","last_peered":"2024-04-22T08:26:28.302571+0000","last_clean":"2024-04-22T08:26:28.302571+0000","last_became_active":"2024-04-22T08:26:15.226139+0000","last_became_peered":"2024-04-22T08:26:15.226139+0000","last_unstale":"2024-04-22T08:26:31.518850+0000","last_undegraded":"2024-04-22T08:26:31.518850+0000","last_fullsized":"2024-04-22T08:26:31.518850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:40.225598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"16'1","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517387+0000","last_change":"2024-04-22T08:26:28.308992+0000","last_active":"2024-04-22T08:26:31.517387+0000","last_peered":"2024-04-22T08:26:31.517387+0000","last_clean":"2024-04-22T08:26:31.517387+0000","last_became_active":"2024-04-22T08:26:28.308543+0000","last_became_peered":"2024-04-22T08:26:28.308543+0000","last_unstale":"2024-04-22T08:26:31.517387+0000","last_undegraded":"2024-04-22T08:26:31.517387+0000","last_fullsized":"2024-04-22T08:26:31.517387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:45.907262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":28173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":139264,"data_stored":36881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:37.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:37.762 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:37.763 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:37.780 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-22T08:26:36.124642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58420,"kb_used_data":284,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685260,"statfs":{"total":193273528320,"available":193213706240,"internally_reserved":0,"allocated":290816,"data_stored":77215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3186,"internal_metadata":59503502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005701"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.526511+0000","last_change":"2024-04-22T08:26:28.305412+0000","last_active":"2024-04-22T08:26:31.526511+0000","last_peered":"2024-04-22T08:26:31.526511+0000","last_clean":"2024-04-22T08:26:31.526511+0000","last_became_active":"2024-04-22T08:26:28.305105+0000","last_became_peered":"2024-04-22T08:26:28.305105+0000","last_unstale":"2024-04-22T08:26:31.526511+0000","last_undegraded":"2024-04-22T08:26:31.526511+0000","last_fullsized":"2024-04-22T08:26:31.526511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:54.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535217+0000","last_change":"2024-04-22T08:26:32.535217+0000","last_active":"2024-04-22T08:26:32.535217+0000","last_peered":"2024-04-22T08:26:32.535217+0000","last_clean":"2024-04-22T08:26:32.535217+0000","last_became_active":"2024-04-22T08:26:32.534389+0000","last_became_peered":"2024-04-22T08:26:32.534389+0000","last_unstale":"2024-04-22T08:26:32.535217+0000","last_undegraded":"2024-04-22T08:26:32.535217+0000","last_fullsized":"2024-04-22T08:26:32.535217+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:22:00.429929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.540729+0000","last_change":"2024-04-22T08:26:32.540729+0000","last_active":"2024-04-22T08:26:32.540729+0000","last_peered":"2024-04-22T08:26:32.540729+0000","last_clean":"2024-04-22T08:26:32.540729+0000","last_became_active":"2024-04-22T08:26:32.539716+0000","last_became_peered":"2024-04-22T08:26:32.539716+0000","last_unstale":"2024-04-22T08:26:32.540729+0000","last_undegraded":"2024-04-22T08:26:32.540729+0000","last_fullsized":"2024-04-22T08:26:32.540729+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:33:11.008804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535237+0000","last_change":"2024-04-22T08:26:32.535237+0000","last_active":"2024-04-22T08:26:32.535237+0000","last_peered":"2024-04-22T08:26:32.535237+0000","last_clean":"2024-04-22T08:26:32.535237+0000","last_became_active":"2024-04-22T08:26:32.534386+0000","last_became_peered":"2024-04-22T08:26:32.534386+0000","last_unstale":"2024-04-22T08:26:32.535237+0000","last_undegraded":"2024-04-22T08:26:32.535237+0000","last_fullsized":"2024-04-22T08:26:32.535237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:27:07.073549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535911+0000","last_change":"2024-04-22T08:26:32.535911+0000","last_active":"2024-04-22T08:26:32.535911+0000","last_peered":"2024-04-22T08:26:32.535911+0000","last_clean":"2024-04-22T08:26:32.535911+0000","last_became_active":"2024-04-22T08:26:32.535336+0000","last_became_peered":"2024-04-22T08:26:32.535336+0000","last_unstale":"2024-04-22T08:26:32.535911+0000","last_undegraded":"2024-04-22T08:26:32.535911+0000","last_fullsized":"2024-04-22T08:26:32.535911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:11:31.743349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517126+0000","last_change":"2024-04-22T08:26:28.308830+0000","last_active":"2024-04-22T08:26:31.517126+0000","last_peered":"2024-04-22T08:26:31.517126+0000","last_clean":"2024-04-22T08:26:31.517126+0000","last_became_active":"2024-04-22T08:26:28.308404+0000","last_became_peered":"2024-04-22T08:26:28.308404+0000","last_unstale":"2024-04-22T08:26:31.517126+0000","last_undegraded":"2024-04-22T08:26:31.517126+0000","last_fullsized":"2024-04-22T08:26:31.517126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:50.451518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.516772+0000","last_change":"2024-04-22T08:26:28.307143+0000","last_active":"2024-04-22T08:26:31.516772+0000","last_peered":"2024-04-22T08:26:31.516772+0000","last_clean":"2024-04-22T08:26:31.516772+0000","last_became_active":"2024-04-22T08:26:28.306710+0000","last_became_peered":"2024-04-22T08:26:28.306710+0000","last_unstale":"2024-04-22T08:26:31.516772+0000","last_undegraded":"2024-04-22T08:26:31.516772+0000","last_fullsized":"2024-04-22T08:26:31.516772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:01.205072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535379+0000","last_change":"2024-04-22T08:26:32.535379+0000","last_active":"2024-04-22T08:26:32.535379+0000","last_peered":"2024-04-22T08:26:32.535379+0000","last_clean":"2024-04-22T08:26:32.535379+0000","last_became_active":"2024-04-22T08:26:32.534533+0000","last_became_peered":"2024-04-22T08:26:32.534533+0000","last_unstale":"2024-04-22T08:26:32.535379+0000","last_undegraded":"2024-04-22T08:26:32.535379+0000","last_fullsized":"2024-04-22T08:26:32.535379+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:50.440072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517739+0000","last_change":"2024-04-22T08:26:15.223179+0000","last_active":"2024-04-22T08:26:31.517739+0000","last_peered":"2024-04-22T08:26:31.517739+0000","last_clean":"2024-04-22T08:26:31.517739+0000","last_became_active":"2024-04-22T08:26:15.222824+0000","last_became_peered":"2024-04-22T08:26:15.222824+0000","last_unstale":"2024-04-22T08:26:31.517739+0000","last_undegraded":"2024-04-22T08:26:31.517739+0000","last_fullsized":"2024-04-22T08:26:31.517739+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:20.128988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.539456+0000","last_change":"2024-04-22T08:26:32.539456+0000","last_active":"2024-04-22T08:26:32.539456+0000","last_peered":"2024-04-22T08:26:32.539456+0000","last_clean":"2024-04-22T08:26:32.539456+0000","last_became_active":"2024-04-22T08:26:32.538651+0000","last_became_peered":"2024-04-22T08:26:32.538651+0000","last_unstale":"2024-04-22T08:26:32.539456+0000","last_undegraded":"2024-04-22T08:26:32.539456+0000","last_fullsized":"2024-04-22T08:26:32.539456+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:58.211326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.535075+0000","last_change":"2024-04-22T08:26:32.535075+0000","last_active":"2024-04-22T08:26:32.535075+0000","last_peered":"2024-04-22T08:26:32.535075+0000","last_clean":"2024-04-22T08:26:32.535075+0000","last_became_active":"2024-04-22T08:26:32.534036+0000","last_became_peered":"2024-04-22T08:26:32.534036+0000","last_unstale":"2024-04-22T08:26:32.535075+0000","last_undegraded":"2024-04-22T08:26:32.535075+0000","last_fullsized":"2024-04-22T08:26:32.535075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:12.320178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.523723+0000","last_change":"2024-04-22T08:26:28.305575+0000","last_active":"2024-04-22T08:26:31.523723+0000","last_peered":"2024-04-22T08:26:31.523723+0000","last_clean":"2024-04-22T08:26:31.523723+0000","last_became_active":"2024-04-22T08:26:28.305202+0000","last_became_peered":"2024-04-22T08:26:28.305202+0000","last_unstale":"2024-04-22T08:26:31.523723+0000","last_undegraded":"2024-04-22T08:26:31.523723+0000","last_fullsized":"2024-04-22T08:26:31.523723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:12.327630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.523814+0000","last_change":"2024-04-22T08:26:15.223614+0000","last_active":"2024-04-22T08:26:31.523814+0000","last_peered":"2024-04-22T08:26:31.523814+0000","last_clean":"2024-04-22T08:26:31.523814+0000","last_became_active":"2024-04-22T08:26:15.223104+0000","last_became_peered":"2024-04-22T08:26:15.223104+0000","last_unstale":"2024-04-22T08:26:31.523814+0000","last_undegraded":"2024-04-22T08:26:31.523814+0000","last_fullsized":"2024-04-22T08:26:31.523814+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:28.144543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.538941+0000","last_change":"2024-04-22T08:26:32.538941+0000","last_active":"2024-04-22T08:26:32.538941+0000","last_peered":"2024-04-22T08:26:32.538941+0000","last_clean":"2024-04-22T08:26:32.538941+0000","last_became_active":"2024-04-22T08:26:32.538143+0000","last_became_peered":"2024-04-22T08:26:32.538143+0000","last_unstale":"2024-04-22T08:26:32.538941+0000","last_undegraded":"2024-04-22T08:26:32.538941+0000","last_fullsized":"2024-04-22T08:26:32.538941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:05:30.643641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.543278+0000","last_change":"2024-04-22T08:26:32.543278+0000","last_active":"2024-04-22T08:26:32.543278+0000","last_peered":"2024-04-22T08:26:32.543278+0000","last_clean":"2024-04-22T08:26:32.543278+0000","last_became_active":"2024-04-22T08:26:32.542460+0000","last_became_peered":"2024-04-22T08:26:32.542460+0000","last_unstale":"2024-04-22T08:26:32.543278+0000","last_undegraded":"2024-04-22T08:26:32.543278+0000","last_fullsized":"2024-04-22T08:26:32.543278+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:13:27.611999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.539100+0000","last_change":"2024-04-22T08:26:32.539100+0000","last_active":"2024-04-22T08:26:32.539100+0000","last_peered":"2024-04-22T08:26:32.539100+0000","last_clean":"2024-04-22T08:26:32.539100+0000","last_became_active":"2024-04-22T08:26:32.538369+0000","last_became_peered":"2024-04-22T08:26:32.538369+0000","last_unstale":"2024-04-22T08:26:32.539100+0000","last_undegraded":"2024-04-22T08:26:32.539100+0000","last_fullsized":"2024-04-22T08:26:32.539100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:54.368834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517613+0000","last_change":"2024-04-22T08:26:28.307119+0000","last_active":"2024-04-22T08:26:31.517613+0000","last_peered":"2024-04-22T08:26:31.517613+0000","last_clean":"2024-04-22T08:26:31.517613+0000","last_became_active":"2024-04-22T08:26:28.306723+0000","last_became_peered":"2024-04-22T08:26:28.306723+0000","last_unstale":"2024-04-22T08:26:31.517613+0000","last_undegraded":"2024-04-22T08:26:31.517613+0000","last_fullsized":"2024-04-22T08:26:31.517613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:08.281247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517610+0000","last_change":"2024-04-22T08:26:15.223815+0000","last_active":"2024-04-22T08:26:31.517610+0000","last_peered":"2024-04-22T08:26:31.517610+0000","last_clean":"2024-04-22T08:26:31.517610+0000","last_became_active":"2024-04-22T08:26:15.223216+0000","last_became_peered":"2024-04-22T08:26:15.223216+0000","last_unstale":"2024-04-22T08:26:31.517610+0000","last_undegraded":"2024-04-22T08:26:31.517610+0000","last_fullsized":"2024-04-22T08:26:31.517610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:53.270991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517948+0000","last_change":"2024-04-22T08:26:28.307051+0000","last_active":"2024-04-22T08:26:31.517948+0000","last_peered":"2024-04-22T08:26:31.517948+0000","last_clean":"2024-04-22T08:26:31.517948+0000","last_became_active":"2024-04-22T08:26:28.306684+0000","last_became_peered":"2024-04-22T08:26:28.306684+0000","last_unstale":"2024-04-22T08:26:31.517948+0000","last_undegraded":"2024-04-22T08:26:31.517948+0000","last_fullsized":"2024-04-22T08:26:31.517948+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:09.266110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.540199+0000","last_change":"2024-04-22T08:26:32.540199+0000","last_active":"2024-04-22T08:26:32.540199+0000","last_peered":"2024-04-22T08:26:32.540199+0000","last_clean":"2024-04-22T08:26:32.540199+0000","last_became_active":"2024-04-22T08:26:32.539618+0000","last_became_peered":"2024-04-22T08:26:32.539618+0000","last_unstale":"2024-04-22T08:26:32.540199+0000","last_undegraded":"2024-04-22T08:26:32.540199+0000","last_fullsized":"2024-04-22T08:26:32.540199+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:03:16.883365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517359+0000","last_change":"2024-04-22T08:26:28.308779+0000","last_active":"2024-04-22T08:26:31.517359+0000","last_peered":"2024-04-22T08:26:31.517359+0000","last_clean":"2024-04-22T08:26:31.517359+0000","last_became_active":"2024-04-22T08:26:28.308412+0000","last_became_peered":"2024-04-22T08:26:28.308412+0000","last_unstale":"2024-04-22T08:26:31.517359+0000","last_undegraded":"2024-04-22T08:26:31.517359+0000","last_fullsized":"2024-04-22T08:26:31.517359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:38.458799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.517947+0000","last_change":"2024-04-22T08:26:15.228695+0000","last_active":"2024-04-22T08:26:31.517947+0000","last_peered":"2024-04-22T08:26:31.517947+0000","last_clean":"2024-04-22T08:26:31.517947+0000","last_became_active":"2024-04-22T08:26:15.227718+0000","last_became_peered":"2024-04-22T08:26:15.227718+0000","last_unstale":"2024-04-22T08:26:31.517947+0000","last_undegraded":"2024-04-22T08:26:31.517947+0000","last_fullsized":"2024-04-22T08:26:31.517947+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:16:35.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T08:26:32.539604+0000","last_change":"2024-04-22T08:26:32.539604+0000","last_active":"2024-04-22T08:26:32.539604+0000","last_peered":"2024-04-22T08:26:32.539604+0000","last_clean":"2024-04-22T08:26:32.539604+0000","last_became_active":"2024-04-22T08:26:32.538756+0000","last_became_peered":"2024-04-22T08:26:32.538756+0000","last_unstale":"2024-04-22T08:26:32.539604+0000","last_undegraded":"2024-04-22T08:26:32.539604+0000","last_fullsized":"2024-04-22T08:26:32.539604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:07:07.923155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"16'1","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T08:26:31.586008+0000","last_change":"2024-04-22T08:26:28.308992+0000","last_active":"2024-04-22T08:26:31.586008+0000","last_peered":"2024-04-22T08:26:31.586008+0000","last_clean":"2024-04-22T08:26:31.586008+0000","last_became_active":"2024-04-22T08:26:28.308543+0000","last_became_peered":"2024-04-22T08:26:28.308543+0000","last_unstale":"2024-04-22T08:26:31.586008+0000","last_undegraded":"2024-04-22T08:26:31.586008+0000","last_fullsized":"2024-04-22T08:26:31.586008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:45.907262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":151552,"data_stored":40334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":139264,"data_stored":36881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:37.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-04-22T08:26:37.782 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-04-22T08:26:37.782 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-04-22T08:26:37.826 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-04-22T08:26:37.826 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-04-22T08:26:37.906 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_1 2024-04-22T08:26:37.907 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_1 2024-04-22T08:26:37.996 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-04-22T08:26:37.996 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-04-22T08:26:38.034 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-04-22T08:26:38.034 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-04-22T08:26:38.115 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_2 2024-04-22T08:26:38.116 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_2 2024-04-22T08:26:38.204 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-04-22T08:26:38.204 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-04-22T08:26:38.243 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-04-22T08:26:38.243 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-04-22T08:26:38.329 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_3 2024-04-22T08:26:38.329 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_3 2024-04-22T08:26:38.407 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-04-22T08:26:38.408 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-04-22T08:26:38.489 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-04-22T08:26:38.490 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-04-22T08:26:38.569 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_4 2024-04-22T08:26:38.570 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_4 2024-04-22T08:26:38.647 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-04-22T08:26:38.647 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-04-22T08:26:38.730 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-04-22T08:26:38.730 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-04-22T08:26:38.808 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_5 2024-04-22T08:26:38.808 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_5 2024-04-22T08:26:38.887 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-04-22T08:26:38.887 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-04-22T08:26:38.973 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-04-22T08:26:38.973 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-04-22T08:26:39.051 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_6 2024-04-22T08:26:39.051 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_6 2024-04-22T08:26:39.132 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-04-22T08:26:39.133 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-04-22T08:26:39.210 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-04-22T08:26:39.210 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-04-22T08:26:39.294 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_7 2024-04-22T08:26:39.294 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_7 2024-04-22T08:26:39.374 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-04-22T08:26:39.374 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-04-22T08:26:39.456 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-04-22T08:26:39.456 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-04-22T08:26:39.535 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_8 2024-04-22T08:26:39.535 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_8 2024-04-22T08:26:39.620 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-04-22T08:26:39.620 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-04-22T08:26:39.702 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-04-22T08:26:39.702 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-04-22T08:26:39.781 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_9 2024-04-22T08:26:39.781 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_9 2024-04-22T08:26:39.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-04-22T08:26:40.053 INFO:teuthology.orchestra.run.smithi049.stdout:{} 2024-04-22T08:26:40.054 INFO:teuthology.orchestra.run.smithi049.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-04-22T08:26:40.070 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T08:26:40.070 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:26:40.153 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T08:26:40.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-04-22T08:26:40.468 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:40.468+0000 7f403222d640 -1 mon.c@2(leader).osd e18 definitely_dead 0 2024-04-22T08:26:40.468 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0 is already down. 2024-04-22T08:26:40.487 INFO:tasks.util.rados:rados -p unfounddel_pool put new_1 /etc/fstab 2024-04-22T08:26:40.488 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_1 /etc/fstab 2024-04-22T08:26:41.210 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-04-22T08:26:41.211 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-04-22T08:26:41.258 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-04-22T08:26:41.258 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-04-22T08:26:41.296 INFO:tasks.util.rados:rados -p unfounddel_pool put new_2 /etc/fstab 2024-04-22T08:26:41.297 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_2 /etc/fstab 2024-04-22T08:26:41.383 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-04-22T08:26:41.383 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-04-22T08:26:41.465 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-04-22T08:26:41.466 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-04-22T08:26:41.547 INFO:tasks.util.rados:rados -p unfounddel_pool put new_3 /etc/fstab 2024-04-22T08:26:41.547 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_3 /etc/fstab 2024-04-22T08:26:41.633 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-04-22T08:26:41.634 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-04-22T08:26:41.719 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-04-22T08:26:41.719 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-04-22T08:26:41.808 INFO:tasks.util.rados:rados -p unfounddel_pool put new_4 /etc/fstab 2024-04-22T08:26:41.808 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_4 /etc/fstab 2024-04-22T08:26:41.847 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-04-22T08:26:41.847 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-04-22T08:26:41.928 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-04-22T08:26:41.929 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-04-22T08:26:42.013 INFO:tasks.util.rados:rados -p unfounddel_pool put new_5 /etc/fstab 2024-04-22T08:26:42.013 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_5 /etc/fstab 2024-04-22T08:26:42.094 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-04-22T08:26:42.094 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-04-22T08:26:42.180 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-04-22T08:26:42.180 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-04-22T08:26:42.220 INFO:tasks.util.rados:rados -p unfounddel_pool put new_6 /etc/fstab 2024-04-22T08:26:42.220 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_6 /etc/fstab 2024-04-22T08:26:42.300 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-04-22T08:26:42.301 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-04-22T08:26:42.389 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-04-22T08:26:42.389 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-04-22T08:26:42.481 INFO:tasks.util.rados:rados -p unfounddel_pool put new_7 /etc/fstab 2024-04-22T08:26:42.481 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_7 /etc/fstab 2024-04-22T08:26:42.519 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-04-22T08:26:42.519 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-04-22T08:26:42.604 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-04-22T08:26:42.604 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-04-22T08:26:42.685 INFO:tasks.util.rados:rados -p unfounddel_pool put new_8 /etc/fstab 2024-04-22T08:26:42.685 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_8 /etc/fstab 2024-04-22T08:26:42.766 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-04-22T08:26:42.766 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-04-22T08:26:42.845 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-04-22T08:26:42.846 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-04-22T08:26:42.924 INFO:tasks.util.rados:rados -p unfounddel_pool put new_9 /etc/fstab 2024-04-22T08:26:42.924 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_9 /etc/fstab 2024-04-22T08:26:43.005 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-04-22T08:26:43.006 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-04-22T08:26:43.086 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-04-22T08:26:43.086 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-04-22T08:26:43.170 INFO:tasks.rep_lost_unfound_delete:osd.0 command_args is foo 2024-04-22T08:26:43.171 INFO:tasks.rep_lost_unfound_delete:() 2024-04-22T08:26:43.173 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T08:26:43.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 --osd-recovery-delay-start 1000 2024-04-22T08:26:43.216 INFO:tasks.ceph.osd.0:Started 2024-04-22T08:26:43.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T08:26:43.329 INFO:teuthology.orchestra.run.smithi049.stderr:no valid command found; 10 closest matches: 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:0 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:1 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:2 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:abort 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:assert 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:config diff 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:config diff get 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:config get 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:config help [] 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:config set ... 2024-04-22T08:26:43.330 INFO:teuthology.orchestra.run.smithi049.stderr:admin_socket: invalid command 2024-04-22T08:26:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T08:26:43.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T08:26:43.817 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:43.816+0000 7ff1b89b17c0 -1 Falling back to public interface 2024-04-22T08:26:47.563 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:47.564+0000 7ff1b89b17c0 -1 osd.0 17 log_to_monitors true 2024-04-22T08:26:48.279 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:48.280+0000 7ff1ac778640 -1 osd.0 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:26:48.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T08:26:48.466 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:26:48.467 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-22T08:26:48.467 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-22T08:26:48.467 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:26:48.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-22T08:26:48.780 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:48.780+0000 7f403222d640 -1 mon.c@2(leader).osd e20 definitely_dead 0 2024-04-22T08:26:48.780 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0 is already in. 2024-04-22T08:26:48.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.0 to be up 2024-04-22T08:26:48.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:49.100 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":20,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:48.260296+0000","last_up_change":"2024-04-22T08:26:40.179246+0000","last_in_change":"2024-04-22T08:26:31.302046+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3.0299999713897705,"score_stable":3.0299999713897705,"optimal_score":0.33000001311302185,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-04-22T08:26:26.331332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":3.0299999713897705,"score_stable":3.0299999713897705,"optimal_score":0.33000001311302185,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":23920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":23920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":23920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":23920}]},"public_addr":"172.21.15.49:6800/23920","cluster_addr":"172.21.15.49:6802/23920","heartbeat_back_addr":"172.21.15.49:6805/23920","heartbeat_front_addr":"172.21.15.49:6804/23920","state":["exists"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":15,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":18},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":15}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:52.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:26:52.426 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:52.426 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":22,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:26:50.285237+0000","last_up_change":"2024-04-22T08:26:49.269319+0000","last_in_change":"2024-04-22T08:26:31.302046+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-04-22T08:26:26.331332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":21,"up_thru":21,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":29827}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":29827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":29827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":29827}]},"public_addr":"172.21.15.49:6800/29827","cluster_addr":"172.21.15.49:6801/29827","heartbeat_back_addr":"172.21.15.49:6803/29827","heartbeat_front_addr":"172.21.15.49:6802/29827","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":15,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":23921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":23921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":23921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":23921}]},"public_addr":"172.21.15.49:6801/23921","cluster_addr":"172.21.15.49:6803/23921","heartbeat_back_addr":"172.21.15.49:6807/23921","heartbeat_front_addr":"172.21.15.49:6806/23921","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":18},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":15}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:26:52.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.0 is up 2024-04-22T08:26:52.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:26:52.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T08:26:52.599 INFO:teuthology.orchestra.run.smithi049.stdout:90194313218 2024-04-22T08:26:52.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:52.602 INFO:teuthology.orchestra.run.smithi049.stdout:34359738381 2024-04-22T08:26:52.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:52.882 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-04-22T08:26:52.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313218 got 1 for osd.0 2024-04-22T08:26:52.924 INFO:teuthology.orchestra.run.smithi049.stdout:34359738380 2024-04-22T08:26:52.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738381 got 34359738380 for osd.1 2024-04-22T08:26:53.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:53.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:54.208 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-04-22T08:26:54.224 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313218 got 1 for osd.0 2024-04-22T08:26:54.234 INFO:teuthology.orchestra.run.smithi049.stdout:34359738380 2024-04-22T08:26:54.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738381 got 34359738380 for osd.1 2024-04-22T08:26:55.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:26:55.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:26:55.529 INFO:teuthology.orchestra.run.smithi049.stdout:90194313219 2024-04-22T08:26:55.547 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313218 got 90194313219 for osd.0 2024-04-22T08:26:55.548 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:55.565 INFO:teuthology.orchestra.run.smithi049.stdout:34359738381 2024-04-22T08:26:55.583 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738381 got 34359738381 for osd.1 2024-04-22T08:26:55.583 DEBUG:teuthology.parallel:result is None 2024-04-22T08:26:55.583 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-04-22T08:26:55.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:26:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:55.920 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:26:55.937 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-22T08:26:54.128987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59108,"kb_used_data":516,"kb_used_omap":5,"kb_used_meta":58554,"kb_avail":188684572,"statfs":{"total":193273528320,"available":193213001728,"internally_reserved":0,"allocated":528384,"data_stored":133096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5991,"internal_metadata":59959449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001704"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.309225+0000","last_change":"2024-04-22T08:26:50.309225+0000","last_active":"2024-04-22T08:26:50.309225+0000","last_peered":"2024-04-22T08:26:50.309225+0000","last_clean":"2024-04-22T08:26:50.309225+0000","last_became_active":"2024-04-22T08:26:50.308548+0000","last_became_peered":"2024-04-22T08:26:50.308548+0000","last_unstale":"2024-04-22T08:26:50.309225+0000","last_undegraded":"2024-04-22T08:26:50.309225+0000","last_fullsized":"2024-04-22T08:26:50.309225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:39:57.370529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'4","reported_seq":24,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.307864+0000","last_change":"2024-04-22T08:26:50.308289+0000","last_active":"2024-04-22T08:26:50.307864+0000","last_peered":"2024-04-22T08:26:50.307864+0000","last_clean":"2024-04-22T08:26:38.491250+0000","last_became_active":"2024-04-22T08:26:50.307614+0000","last_became_peered":"2024-04-22T08:26:50.307614+0000","last_unstale":"2024-04-22T08:26:50.307864+0000","last_undegraded":"2024-04-22T08:26:50.290191+0000","last_fullsized":"2024-04-22T08:26:50.307864+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:54:17.484080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'3","reported_seq":42,"reported_epoch":22,"state":"active+recovering+degraded","last_fresh":"2024-04-22T08:26:50.307993+0000","last_change":"2024-04-22T08:26:50.318720+0000","last_active":"2024-04-22T08:26:50.307993+0000","last_peered":"2024-04-22T08:26:50.307993+0000","last_clean":"2024-04-22T08:26:39.376175+0000","last_became_active":"2024-04-22T08:26:50.307838+0000","last_became_peered":"2024-04-22T08:26:50.307838+0000","last_unstale":"2024-04-22T08:26:50.307993+0000","last_undegraded":"2024-04-22T08:26:50.293612+0000","last_fullsized":"2024-04-22T08:26:50.307993+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:07:25.906341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.307903+0000","last_change":"2024-04-22T08:26:50.307903+0000","last_active":"2024-04-22T08:26:50.307903+0000","last_peered":"2024-04-22T08:26:50.307903+0000","last_clean":"2024-04-22T08:26:50.307903+0000","last_became_active":"2024-04-22T08:26:50.307151+0000","last_became_peered":"2024-04-22T08:26:50.307151+0000","last_unstale":"2024-04-22T08:26:50.307903+0000","last_undegraded":"2024-04-22T08:26:50.307903+0000","last_fullsized":"2024-04-22T08:26:50.307903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:44.336121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"19'5","reported_seq":34,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.305927+0000","last_change":"2024-04-22T08:26:50.306352+0000","last_active":"2024-04-22T08:26:50.305927+0000","last_peered":"2024-04-22T08:26:50.305927+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:26:50.305695+0000","last_became_peered":"2024-04-22T08:26:50.305695+0000","last_unstale":"2024-04-22T08:26:50.305927+0000","last_undegraded":"2024-04-22T08:26:50.289641+0000","last_fullsized":"2024-04-22T08:26:50.305927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:31.307831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"19'3","reported_seq":32,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.310664+0000","last_change":"2024-04-22T08:26:50.311984+0000","last_active":"2024-04-22T08:26:50.310664+0000","last_peered":"2024-04-22T08:26:50.310664+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:26:50.310437+0000","last_became_peered":"2024-04-22T08:26:50.310437+0000","last_unstale":"2024-04-22T08:26:50.310664+0000","last_undegraded":"2024-04-22T08:26:50.295654+0000","last_fullsized":"2024-04-22T08:26:50.310664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:02:08.028767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"19'4","reported_seq":33,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.308379+0000","last_change":"2024-04-22T08:26:50.308806+0000","last_active":"2024-04-22T08:26:50.308379+0000","last_peered":"2024-04-22T08:26:50.308379+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:26:50.308157+0000","last_became_peered":"2024-04-22T08:26:50.308157+0000","last_unstale":"2024-04-22T08:26:50.308379+0000","last_undegraded":"2024-04-22T08:26:50.293269+0000","last_fullsized":"2024-04-22T08:26:50.308379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:06:13.832877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"19'1","reported_seq":32,"reported_epoch":22,"state":"active+recovering+degraded","last_fresh":"2024-04-22T08:26:50.305101+0000","last_change":"2024-04-22T08:26:50.318081+0000","last_active":"2024-04-22T08:26:50.305101+0000","last_peered":"2024-04-22T08:26:50.305101+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:26:50.304882+0000","last_became_peered":"2024-04-22T08:26:50.304882+0000","last_unstale":"2024-04-22T08:26:50.305101+0000","last_undegraded":"2024-04-22T08:26:50.289383+0000","last_fullsized":"2024-04-22T08:26:50.305101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:43.370821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.307800+0000","last_change":"2024-04-22T08:26:50.307800+0000","last_active":"2024-04-22T08:26:50.307800+0000","last_peered":"2024-04-22T08:26:50.307800+0000","last_clean":"2024-04-22T08:26:50.307800+0000","last_became_active":"2024-04-22T08:26:50.307517+0000","last_became_peered":"2024-04-22T08:26:50.307517+0000","last_unstale":"2024-04-22T08:26:50.307800+0000","last_undegraded":"2024-04-22T08:26:50.307800+0000","last_fullsized":"2024-04-22T08:26:50.307800+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:07.436301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"19'5","reported_seq":43,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.309722+0000","last_change":"2024-04-22T08:26:50.313377+0000","last_active":"2024-04-22T08:26:50.309722+0000","last_peered":"2024-04-22T08:26:50.309722+0000","last_clean":"2024-04-22T08:26:39.458282+0000","last_became_active":"2024-04-22T08:26:50.309494+0000","last_became_peered":"2024-04-22T08:26:50.309494+0000","last_unstale":"2024-04-22T08:26:50.309722+0000","last_undegraded":"2024-04-22T08:26:50.294684+0000","last_fullsized":"2024-04-22T08:26:50.309722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:26:52.338976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.305392+0000","last_change":"2024-04-22T08:26:50.305392+0000","last_active":"2024-04-22T08:26:50.305392+0000","last_peered":"2024-04-22T08:26:50.305392+0000","last_clean":"2024-04-22T08:26:50.305392+0000","last_became_active":"2024-04-22T08:26:50.305118+0000","last_became_peered":"2024-04-22T08:26:50.305118+0000","last_unstale":"2024-04-22T08:26:50.305392+0000","last_undegraded":"2024-04-22T08:26:50.305392+0000","last_fullsized":"2024-04-22T08:26:50.305392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:01.081556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'6","reported_seq":26,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.307015+0000","last_change":"2024-04-22T08:26:50.307126+0000","last_active":"2024-04-22T08:26:50.307015+0000","last_peered":"2024-04-22T08:26:50.307015+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:26:50.306926+0000","last_became_peered":"2024-04-22T08:26:50.306926+0000","last_unstale":"2024-04-22T08:26:50.307015+0000","last_undegraded":"2024-04-22T08:26:50.291177+0000","last_fullsized":"2024-04-22T08:26:50.307015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:35:47.309650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.306926+0000","last_change":"2024-04-22T08:26:50.306926+0000","last_active":"2024-04-22T08:26:50.306926+0000","last_peered":"2024-04-22T08:26:50.306926+0000","last_clean":"2024-04-22T08:26:50.306926+0000","last_became_active":"2024-04-22T08:26:50.306652+0000","last_became_peered":"2024-04-22T08:26:50.306652+0000","last_unstale":"2024-04-22T08:26:50.306926+0000","last_undegraded":"2024-04-22T08:26:50.306926+0000","last_fullsized":"2024-04-22T08:26:50.306926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:58.378092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"19'3","reported_seq":33,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.311255+0000","last_change":"2024-04-22T08:26:50.311852+0000","last_active":"2024-04-22T08:26:50.311255+0000","last_peered":"2024-04-22T08:26:50.311255+0000","last_clean":"2024-04-22T08:26:38.244414+0000","last_became_active":"2024-04-22T08:26:50.311033+0000","last_became_peered":"2024-04-22T08:26:50.311033+0000","last_unstale":"2024-04-22T08:26:50.311255+0000","last_undegraded":"2024-04-22T08:26:50.296192+0000","last_fullsized":"2024-04-22T08:26:50.311255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:37.540048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.308718+0000","last_change":"2024-04-22T08:26:50.308718+0000","last_active":"2024-04-22T08:26:50.308718+0000","last_peered":"2024-04-22T08:26:50.308718+0000","last_clean":"2024-04-22T08:26:50.308718+0000","last_became_active":"2024-04-22T08:26:50.307984+0000","last_became_peered":"2024-04-22T08:26:50.307984+0000","last_unstale":"2024-04-22T08:26:50.308718+0000","last_undegraded":"2024-04-22T08:26:50.308718+0000","last_fullsized":"2024-04-22T08:26:50.308718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:12:05.599968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"19'4","reported_seq":34,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.309772+0000","last_change":"2024-04-22T08:26:50.310027+0000","last_active":"2024-04-22T08:26:50.309772+0000","last_peered":"2024-04-22T08:26:50.309772+0000","last_clean":"2024-04-22T08:26:38.888837+0000","last_became_active":"2024-04-22T08:26:50.309693+0000","last_became_peered":"2024-04-22T08:26:50.309693+0000","last_unstale":"2024-04-22T08:26:50.309772+0000","last_undegraded":"2024-04-22T08:26:50.295616+0000","last_fullsized":"2024-04-22T08:26:50.309772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:04:59.760782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"19'1","reported_seq":29,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.308413+0000","last_change":"2024-04-22T08:26:50.308572+0000","last_active":"2024-04-22T08:26:50.308413+0000","last_peered":"2024-04-22T08:26:50.308413+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:26:50.308338+0000","last_became_peered":"2024-04-22T08:26:50.308338+0000","last_unstale":"2024-04-22T08:26:50.308413+0000","last_undegraded":"2024-04-22T08:26:50.293620+0000","last_fullsized":"2024-04-22T08:26:50.308413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:31.461565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.309776+0000","last_change":"2024-04-22T08:26:50.309776+0000","last_active":"2024-04-22T08:26:50.309776+0000","last_peered":"2024-04-22T08:26:50.309776+0000","last_clean":"2024-04-22T08:26:50.309776+0000","last_became_active":"2024-04-22T08:26:50.308987+0000","last_became_peered":"2024-04-22T08:26:50.308987+0000","last_unstale":"2024-04-22T08:26:50.309776+0000","last_undegraded":"2024-04-22T08:26:50.309776+0000","last_fullsized":"2024-04-22T08:26:50.309776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:02:24.626725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'5","reported_seq":34,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.310303+0000","last_change":"2024-04-22T08:26:50.311813+0000","last_active":"2024-04-22T08:26:50.310303+0000","last_peered":"2024-04-22T08:26:50.310303+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:26:50.310058+0000","last_became_peered":"2024-04-22T08:26:50.310058+0000","last_unstale":"2024-04-22T08:26:50.310303+0000","last_undegraded":"2024-04-22T08:26:50.294244+0000","last_fullsized":"2024-04-22T08:26:50.310303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:33:59.440758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.305753+0000","last_change":"2024-04-22T08:26:50.305753+0000","last_active":"2024-04-22T08:26:50.305753+0000","last_peered":"2024-04-22T08:26:50.305753+0000","last_clean":"2024-04-22T08:26:50.305753+0000","last_became_active":"2024-04-22T08:26:50.305348+0000","last_became_peered":"2024-04-22T08:26:50.305348+0000","last_unstale":"2024-04-22T08:26:50.305753+0000","last_undegraded":"2024-04-22T08:26:50.305753+0000","last_fullsized":"2024-04-22T08:26:50.305753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:58.202170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"19'2","reported_seq":30,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.308452+0000","last_change":"2024-04-22T08:26:50.308600+0000","last_active":"2024-04-22T08:26:50.308452+0000","last_peered":"2024-04-22T08:26:50.308452+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:26:50.308375+0000","last_became_peered":"2024-04-22T08:26:50.308375+0000","last_unstale":"2024-04-22T08:26:50.308452+0000","last_undegraded":"2024-04-22T08:26:50.294291+0000","last_fullsized":"2024-04-22T08:26:50.308452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:38.246579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.311962+0000","last_change":"2024-04-22T08:26:50.311962+0000","last_active":"2024-04-22T08:26:50.311962+0000","last_peered":"2024-04-22T08:26:50.311962+0000","last_clean":"2024-04-22T08:26:50.311962+0000","last_became_active":"2024-04-22T08:26:50.311076+0000","last_became_peered":"2024-04-22T08:26:50.311076+0000","last_unstale":"2024-04-22T08:26:50.311962+0000","last_undegraded":"2024-04-22T08:26:50.311962+0000","last_fullsized":"2024-04-22T08:26:50.311962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:26:52.249178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-22T08:26:50.310252+0000","last_change":"2024-04-22T08:26:50.310252+0000","last_active":"2024-04-22T08:26:50.310252+0000","last_peered":"2024-04-22T08:26:50.310252+0000","last_clean":"2024-04-22T08:26:50.310252+0000","last_became_active":"2024-04-22T08:26:50.309457+0000","last_became_peered":"2024-04-22T08:26:50.309457+0000","last_unstale":"2024-04-22T08:26:50.310252+0000","last_undegraded":"2024-04-22T08:26:50.310252+0000","last_fullsized":"2024-04-22T08:26:50.310252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:35.282218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'9","reported_seq":38,"reported_epoch":22,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:26:50.310799+0000","last_change":"2024-04-22T08:26:50.311609+0000","last_active":"2024-04-22T08:26:50.310799+0000","last_peered":"2024-04-22T08:26:50.310799+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:26:50.310570+0000","last_became_peered":"2024-04-22T08:26:50.310570+0000","last_unstale":"2024-04-22T08:26:50.310799+0000","last_undegraded":"2024-04-22T08:26:50.295802+0000","last_fullsized":"2024-04-22T08:26:50.310799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:07:50.997342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":303104,"data_stored":72469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313219,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":220,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":225280,"data_stored":60627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":32304851},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:26:55.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:26:56.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:26:56.283 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27/56 objects degraded (48.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":51,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":22,"num_osds":3,"num_up_osds":2,"osd_up_since":1713774409,"num_in_osds":2,"osd_in_since":1713774391,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+degraded","count":12},{"state_name":"active+clean","count":10},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":24,"num_pools":2,"num_objects":28,"data_bytes":17276,"bytes_used":60526592,"bytes_avail":193213001728,"bytes_total":193273528320,"degraded_objects":27,"degraded_total":56,"degraded_ratio":0.48214285714285715},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-22T08:26:05.083154+0000","services":{}},"progress_events":{}} 2024-04-22T08:26:56.298 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '7ee9519b-73a7-4b76-a33e-63cd80692d01', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 27/56 objects degraded (48.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 12, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 51, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 22, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1713774409, 'num_in_osds': 2, 'osd_in_since': 1713774391, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+degraded', 'count': 12}, {'state_name': 'active+clean', 'count': 10}, {'state_name': 'active+recovering+degraded', 'count': 2}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 28, 'data_bytes': 17276, 'bytes_used': 60526592, 'bytes_avail': 193213001728, 'bytes_total': 193273528320, 'degraded_objects': 27, 'degraded_total': 56, 'degraded_ratio': 0.48214285714285715}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-04-22T08:26:05.083154+0000', 'services': {}}, 'progress_events': {}} 2024-04-22T08:26:56.299 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-04-22T08:26:56.299 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T08:26:56.299 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:26:56.381 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T08:26:56.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-04-22T08:26:56.674 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:56.672+0000 7f403222d640 -1 mon.c@2(leader).osd e22 definitely_dead 0 2024-04-22T08:26:57.321 INFO:teuthology.orchestra.run.smithi049.stderr:marked down osd.1. 2024-04-22T08:26:57.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-04-22T08:26:57.657 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:26:57.656+0000 7f403222d640 -1 mon.c@2(leader).osd e23 definitely_dead 0 2024-04-22T08:26:58.321 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.1. 2024-04-22T08:26:58.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-04-22T08:26:59.348 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.344+0000 7ff195f3f640 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.348 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff199746640 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff199f47640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff19573e640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff194f3d640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff198744640 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff194f3d640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff198744640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-04-22T08:26:59.349 INFO:teuthology.orchestra.run.smithi049.stderr:marked osd lost in epoch 23 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff193f3b640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff197742640 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff19373a640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff192f39640 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff196f41640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-04-22T08:26:59.350 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:26:59.348+0000 7ff196740640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-04-22T08:26:59.370 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T08:26:59.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-22T08:26:59.372 INFO:tasks.ceph.osd.2:Started 2024-04-22T08:26:59.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:no valid command found; 10 closest matches: 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:0 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:1 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:2 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:abort 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:assert 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:config diff 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:config diff get 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:config get 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:config help [] 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:config set ... 2024-04-22T08:26:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:admin_socket: invalid command 2024-04-22T08:26:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T08:26:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T08:26:59.972 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:26:59.972+0000 7f04a41837c0 -1 Falling back to public interface 2024-04-22T08:27:03.729 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:27:03.728+0000 7f04a41837c0 -1 osd.2 14 log_to_monitors true 2024-04-22T08:27:04.364 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff199f47640 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.364 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff195f3f640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-04-22T08:27:04.364 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff199746640 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.365 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff194f3d640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-04-22T08:27:04.365 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff197f43640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-04-22T08:27:04.365 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff199746640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.366 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff198f45640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.366 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff19373a640 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-04-22T08:27:04.366 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff198744640 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.367 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff196740640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-04-22T08:27:04.367 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff19373a640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-04-22T08:27:04.367 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.364+0000 7ff198744640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-04-22T08:27:04.367 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.368+0000 7ff192f39640 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-04-22T08:27:04.367 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-04-22T08:27:04.368+0000 7ff196f41640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-04-22T08:27:04.373 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-04-22T08:27:04.372+0000 7f049bf52640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:27:04.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T08:27:04.631 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-22T08:27:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-22T08:27:04.631 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:04.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-04-22T08:27:04.954 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:27:04.952+0000 7f403222d640 -1 mon.c@2(leader).osd e26 definitely_dead 0 2024-04-22T08:27:05.372 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.2. 2024-04-22T08:27:05.398 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.2 to be up 2024-04-22T08:27:05.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:27:05.693 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:05.693 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:27:05.363774+0000","last_up_change":"2024-04-22T08:27:05.363774+0000","last_in_change":"2024-04-22T08:27:05.363774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.4900000095367432,"score_stable":1.4900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-04-22T08:26:26.331332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":21,"up_thru":23,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":29827}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":29827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":29827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":29827}]},"public_addr":"172.21.15.49:6800/29827","cluster_addr":"172.21.15.49:6801/29827","heartbeat_back_addr":"172.21.15.49:6803/29827","heartbeat_front_addr":"172.21.15.49:6802/29827","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":21,"down_at":23,"lost_at":23,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":23922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":23922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":23922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":23922}]},"public_addr":"172.21.15.49:6808/23922","cluster_addr":"172.21.15.49:6809/23922","heartbeat_back_addr":"172.21.15.49:6811/23922","heartbeat_front_addr":"172.21.15.49:6810/23922","state":["exists"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":14,"up_from":27,"up_thru":12,"down_at":15,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":31087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":31087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":31087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":31087}]},"public_addr":"172.21.15.49:6804/31087","cluster_addr":"172.21.15.49:6805/31087","heartbeat_back_addr":"172.21.15.49:6807/31087","heartbeat_front_addr":"172.21.15.49:6806/31087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":18},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":15}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:27:05.711 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.2 is up 2024-04-22T08:27:05.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:27:05.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:27:05.861 INFO:teuthology.orchestra.run.smithi049.stdout:115964116994 2024-04-22T08:27:05.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:27:05.871 INFO:teuthology.orchestra.run.smithi049.stdout:90194313222 2024-04-22T08:27:05.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:27:06.142 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T08:27:06.159 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 115964116994 got 0 for osd.2 2024-04-22T08:27:06.197 INFO:teuthology.orchestra.run.smithi049.stdout:90194313221 2024-04-22T08:27:06.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313222 got 90194313221 for osd.0 2024-04-22T08:27:07.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:27:07.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:27:07.471 INFO:teuthology.orchestra.run.smithi049.stdout:115964116994 2024-04-22T08:27:07.490 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 115964116994 got 115964116994 for osd.2 2024-04-22T08:27:07.490 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:07.509 INFO:teuthology.orchestra.run.smithi049.stdout:90194313222 2024-04-22T08:27:07.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313222 got 90194313222 for osd.0 2024-04-22T08:27:07.527 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:07.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-04-22T08:27:07.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:07.863 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:07.863 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:07.879 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-22T08:27:06.375253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63680,"kb_used_data":480,"kb_used_omap":8,"kb_used_meta":63159,"kb_avail":188680000,"statfs":{"total":193273528320,"available":193208320000,"internally_reserved":0,"allocated":491520,"data_stored":145894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":64675234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":20,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.244716"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":32,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.386561+0000","last_change":"2024-04-22T08:27:05.386733+0000","last_active":"2024-04-22T08:27:05.386561+0000","last_peered":"2024-04-22T08:27:04.371030+0000","last_clean":"2024-04-22T08:26:50.313731+0000","last_became_active":"2024-04-22T08:26:58.339769+0000","last_became_peered":"2024-04-22T08:26:58.339769+0000","last_unstale":"2024-04-22T08:27:05.386561+0000","last_undegraded":"2024-04-22T08:27:05.386561+0000","last_fullsized":"2024-04-22T08:27:05.386561+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:07.136723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'4","reported_seq":13,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:04.371847+0000","last_change":"2024-04-22T08:27:05.417257+0000","last_active":"2024-04-22T08:27:05.417257+0000","last_peered":"2024-04-22T08:27:04.371847+0000","last_clean":"2024-04-22T08:26:38.491250+0000","last_became_active":"2024-04-22T08:26:58.337139+0000","last_became_peered":"2024-04-22T08:26:58.337139+0000","last_unstale":"2024-04-22T08:27:04.371847+0000","last_undegraded":"2024-04-22T08:26:58.331968+0000","last_fullsized":"2024-04-22T08:26:58.331700+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:25:59.497393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"19'3","reported_seq":6,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.417986+0000","last_change":"2024-04-22T08:27:05.417986+0000","last_active":"2024-04-22T08:27:05.417986+0000","last_peered":"2024-04-22T08:27:04.370437+0000","last_clean":"2024-04-22T08:26:39.376175+0000","last_became_active":"2024-04-22T08:26:58.339793+0000","last_became_peered":"2024-04-22T08:26:58.339793+0000","last_unstale":"2024-04-22T08:27:05.417986+0000","last_undegraded":"2024-04-22T08:27:05.417986+0000","last_fullsized":"2024-04-22T08:27:05.417986+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:01.624202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.412716+0000","last_change":"2024-04-22T08:27:04.375038+0000","last_active":"2024-04-22T08:27:04.375038+0000","last_peered":"2024-04-22T08:26:28.296126+0000","last_clean":"2024-04-22T08:26:28.296126+0000","last_became_active":"2024-04-22T08:26:28.295702+0000","last_became_peered":"2024-04-22T08:26:28.295702+0000","last_unstale":"2024-04-22T08:27:05.412716+0000","last_undegraded":"2024-04-22T08:27:05.412716+0000","last_fullsized":"2024-04-22T08:27:05.412716+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:50.414600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.379254+0000","last_change":"2024-04-22T08:27:05.383889+0000","last_active":"2024-04-22T08:27:05.379254+0000","last_peered":"2024-04-22T08:27:04.371101+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:26:58.336648+0000","last_became_peered":"2024-04-22T08:26:58.336648+0000","last_unstale":"2024-04-22T08:27:05.379254+0000","last_undegraded":"2024-04-22T08:27:05.379254+0000","last_fullsized":"2024-04-22T08:27:05.379254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:03.153469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"19'3","reported_seq":38,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.381294+0000","last_change":"2024-04-22T08:27:05.382761+0000","last_active":"2024-04-22T08:27:05.381294+0000","last_peered":"2024-04-22T08:27:04.369133+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:26:58.331334+0000","last_became_peered":"2024-04-22T08:26:58.331334+0000","last_unstale":"2024-04-22T08:27:05.381294+0000","last_undegraded":"2024-04-22T08:27:05.381294+0000","last_fullsized":"2024-04-22T08:27:05.381294+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:44.515619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"19'4","reported_seq":39,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.387638+0000","last_change":"2024-04-22T08:27:05.387802+0000","last_active":"2024-04-22T08:27:05.387638+0000","last_peered":"2024-04-22T08:27:04.369413+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:26:58.331213+0000","last_became_peered":"2024-04-22T08:26:58.331213+0000","last_unstale":"2024-04-22T08:27:05.387638+0000","last_undegraded":"2024-04-22T08:27:05.387638+0000","last_fullsized":"2024-04-22T08:27:05.387638+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:42.611003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"19'1","reported_seq":43,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.379207+0000","last_change":"2024-04-22T08:27:05.379684+0000","last_active":"2024-04-22T08:27:05.379207+0000","last_peered":"2024-04-22T08:27:04.368766+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:26:58.329200+0000","last_became_peered":"2024-04-22T08:26:58.329200+0000","last_unstale":"2024-04-22T08:27:05.379207+0000","last_undegraded":"2024-04-22T08:27:05.379207+0000","last_fullsized":"2024-04-22T08:27:05.379207+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:29.420943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":37,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.380212+0000","last_change":"2024-04-22T08:27:05.380590+0000","last_active":"2024-04-22T08:27:05.380212+0000","last_peered":"2024-04-22T08:27:04.368267+0000","last_clean":"2024-04-22T08:26:32.529717+0000","last_became_active":"2024-04-22T08:26:58.328192+0000","last_became_peered":"2024-04-22T08:26:58.328192+0000","last_unstale":"2024-04-22T08:27:05.380212+0000","last_undegraded":"2024-04-22T08:27:05.380212+0000","last_fullsized":"2024-04-22T08:27:05.380212+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:24.390610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"19'5","reported_seq":6,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.417631+0000","last_change":"2024-04-22T08:27:05.417631+0000","last_active":"2024-04-22T08:27:05.417631+0000","last_peered":"2024-04-22T08:27:04.370591+0000","last_clean":"2024-04-22T08:26:39.458282+0000","last_became_active":"2024-04-22T08:26:58.339672+0000","last_became_peered":"2024-04-22T08:26:58.339672+0000","last_unstale":"2024-04-22T08:27:05.417631+0000","last_undegraded":"2024-04-22T08:27:05.417631+0000","last_fullsized":"2024-04-22T08:27:05.417631+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:21.084393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.411922+0000","last_change":"2024-04-22T08:27:04.375289+0000","last_active":"2024-04-22T08:27:04.375289+0000","last_peered":"2024-04-22T08:26:15.216982+0000","last_clean":"2024-04-22T08:26:15.216982+0000","last_became_active":"2024-04-22T08:26:15.216171+0000","last_became_peered":"2024-04-22T08:26:15.216171+0000","last_unstale":"2024-04-22T08:27:05.411922+0000","last_undegraded":"2024-04-22T08:27:05.411922+0000","last_fullsized":"2024-04-22T08:27:05.411922+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:17:56.512357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'6","reported_seq":37,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.380921+0000","last_change":"2024-04-22T08:27:05.383234+0000","last_active":"2024-04-22T08:27:05.380921+0000","last_peered":"2024-04-22T08:27:04.369696+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:26:58.331824+0000","last_became_peered":"2024-04-22T08:26:58.331824+0000","last_unstale":"2024-04-22T08:27:05.380921+0000","last_undegraded":"2024-04-22T08:27:05.380921+0000","last_fullsized":"2024-04-22T08:27:05.380921+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:20:05.835129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.379484+0000","last_change":"2024-04-22T08:27:05.379864+0000","last_active":"2024-04-22T08:27:05.379484+0000","last_peered":"2024-04-22T08:27:04.368648+0000","last_clean":"2024-04-22T08:26:50.312762+0000","last_became_active":"2024-04-22T08:26:58.328545+0000","last_became_peered":"2024-04-22T08:26:58.328545+0000","last_unstale":"2024-04-22T08:27:05.379484+0000","last_undegraded":"2024-04-22T08:27:05.379484+0000","last_fullsized":"2024-04-22T08:27:05.379484+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:51:26.920968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"19'3","reported_seq":13,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:04.368790+0000","last_change":"2024-04-22T08:27:05.417226+0000","last_active":"2024-04-22T08:27:05.417226+0000","last_peered":"2024-04-22T08:27:04.368790+0000","last_clean":"2024-04-22T08:26:38.244414+0000","last_became_active":"2024-04-22T08:26:58.328733+0000","last_became_peered":"2024-04-22T08:26:58.328733+0000","last_unstale":"2024-04-22T08:27:04.368790+0000","last_undegraded":"2024-04-22T08:26:58.327259+0000","last_fullsized":"2024-04-22T08:26:58.326660+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:58.882515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":5,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:04.370129+0000","last_change":"2024-04-22T08:27:05.851527+0000","last_active":"2024-04-22T08:27:05.851527+0000","last_peered":"2024-04-22T08:27:04.370129+0000","last_clean":"2024-04-22T08:26:32.544295+0000","last_became_active":"2024-04-22T08:26:58.328720+0000","last_became_peered":"2024-04-22T08:26:58.328720+0000","last_unstale":"2024-04-22T08:27:04.370129+0000","last_undegraded":"2024-04-22T08:27:04.370129+0000","last_fullsized":"2024-04-22T08:26:58.326875+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"19'4","reported_seq":13,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:04.368435+0000","last_change":"2024-04-22T08:27:05.413958+0000","last_active":"2024-04-22T08:27:05.413958+0000","last_peered":"2024-04-22T08:27:04.368435+0000","last_clean":"2024-04-22T08:26:38.888837+0000","last_became_active":"2024-04-22T08:26:58.335621+0000","last_became_peered":"2024-04-22T08:26:58.335621+0000","last_unstale":"2024-04-22T08:27:04.368435+0000","last_undegraded":"2024-04-22T08:26:58.328489+0000","last_fullsized":"2024-04-22T08:26:58.326640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:48:11.903425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"19'1","reported_seq":35,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.386637+0000","last_change":"2024-04-22T08:27:05.386801+0000","last_active":"2024-04-22T08:27:05.386637+0000","last_peered":"2024-04-22T08:27:04.370634+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:26:58.336648+0000","last_became_peered":"2024-04-22T08:26:58.336648+0000","last_unstale":"2024-04-22T08:27:05.386637+0000","last_undegraded":"2024-04-22T08:27:05.386637+0000","last_fullsized":"2024-04-22T08:27:05.386637+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:16:18.018588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.382564+0000","last_change":"2024-04-22T08:27:05.382714+0000","last_active":"2024-04-22T08:27:05.382564+0000","last_peered":"2024-04-22T08:27:04.369590+0000","last_clean":"2024-04-22T08:26:32.531766+0000","last_became_active":"2024-04-22T08:26:58.336625+0000","last_became_peered":"2024-04-22T08:26:58.336625+0000","last_unstale":"2024-04-22T08:27:05.382564+0000","last_undegraded":"2024-04-22T08:27:05.382564+0000","last_fullsized":"2024-04-22T08:27:05.382564+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:53.750534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"19'5","reported_seq":40,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.382940+0000","last_change":"2024-04-22T08:27:05.383044+0000","last_active":"2024-04-22T08:27:05.382940+0000","last_peered":"2024-04-22T08:27:04.371896+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:26:58.336715+0000","last_became_peered":"2024-04-22T08:26:58.336715+0000","last_unstale":"2024-04-22T08:27:05.382940+0000","last_undegraded":"2024-04-22T08:27:05.382940+0000","last_fullsized":"2024-04-22T08:27:05.382940+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:58.847587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.412168+0000","last_change":"2024-04-22T08:27:04.375685+0000","last_active":"2024-04-22T08:27:04.375685+0000","last_peered":"2024-04-22T08:26:15.217054+0000","last_clean":"2024-04-22T08:26:15.217054+0000","last_became_active":"2024-04-22T08:26:15.216307+0000","last_became_peered":"2024-04-22T08:26:15.216307+0000","last_unstale":"2024-04-22T08:27:05.412168+0000","last_undegraded":"2024-04-22T08:27:05.412168+0000","last_fullsized":"2024-04-22T08:27:05.412168+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:11.049256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"19'2","reported_seq":36,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.386817+0000","last_change":"2024-04-22T08:27:05.387245+0000","last_active":"2024-04-22T08:27:05.386817+0000","last_peered":"2024-04-22T08:27:04.370966+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:26:58.341922+0000","last_became_peered":"2024-04-22T08:26:58.341922+0000","last_unstale":"2024-04-22T08:27:05.386817+0000","last_undegraded":"2024-04-22T08:27:05.386817+0000","last_fullsized":"2024-04-22T08:27:05.386817+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:00.973403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.383943+0000","last_change":"2024-04-22T08:27:05.384103+0000","last_active":"2024-04-22T08:27:05.383943+0000","last_peered":"2024-04-22T08:27:04.371384+0000","last_clean":"2024-04-22T08:26:32.531190+0000","last_became_active":"2024-04-22T08:26:58.336415+0000","last_became_peered":"2024-04-22T08:26:58.336415+0000","last_unstale":"2024-04-22T08:27:05.383943+0000","last_undegraded":"2024-04-22T08:27:05.383943+0000","last_fullsized":"2024-04-22T08:27:05.383943+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:43:03.022816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":5,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:04.370175+0000","last_change":"2024-04-22T08:27:05.851584+0000","last_active":"2024-04-22T08:27:05.851584+0000","last_peered":"2024-04-22T08:27:04.370175+0000","last_clean":"2024-04-22T08:26:32.541815+0000","last_became_active":"2024-04-22T08:26:58.341190+0000","last_became_peered":"2024-04-22T08:26:58.341190+0000","last_unstale":"2024-04-22T08:27:04.370175+0000","last_undegraded":"2024-04-22T08:27:04.370175+0000","last_fullsized":"2024-04-22T08:26:58.329376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"19'9","reported_seq":44,"reported_epoch":27,"state":"peering","last_fresh":"2024-04-22T08:27:05.382813+0000","last_change":"2024-04-22T08:27:05.383043+0000","last_active":"2024-04-22T08:27:05.382813+0000","last_peered":"2024-04-22T08:27:04.369356+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:26:58.340750+0000","last_became_peered":"2024-04-22T08:26:58.340750+0000","last_unstale":"2024-04-22T08:27:05.382813+0000","last_undegraded":"2024-04-22T08:27:05.382813+0000","last_fullsized":"2024-04-22T08:27:05.382813+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:35:04.485078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964116994,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":224,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":229376,"data_stored":70727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":32370383},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313222,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":256,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":262144,"data_stored":75167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":32304851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:07.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:27:08.217 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:08.218 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":63,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":28,"num_osds":3,"num_up_osds":2,"osd_up_since":1713774425,"num_in_osds":2,"osd_in_since":1713774425,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":24}],"num_pgs":24,"num_pools":2,"num_objects":28,"data_bytes":17276,"bytes_used":65208320,"bytes_avail":193208320000,"bytes_total":193273528320,"inactive_pgs_ratio":1},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-22T08:26:05.083154+0000","services":{}},"progress_events":{}} 2024-04-22T08:27:08.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '7ee9519b-73a7-4b76-a33e-63cd80692d01', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 12, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 63, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1713774425, 'num_in_osds': 2, 'osd_in_since': 1713774425, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 24}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 28, 'data_bytes': 17276, 'bytes_used': 65208320, 'bytes_avail': 193208320000, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 1}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-04-22T08:26:05.083154+0000', 'services': {}}, 'progress_events': {}} 2024-04-22T08:27:10.180 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:27:10.180+0000 7f4034a32640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-04-22T08:27:11.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:11.548 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:11.558 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-04-22T08:27:10.133389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63696,"kb_used_data":496,"kb_used_omap":8,"kb_used_meta":63159,"kb_avail":188679984,"statfs":{"total":193273528320,"available":193208303616,"internally_reserved":0,"allocated":507904,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":64675234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001839"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404664+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:06.404664+0000","last_peered":"2024-04-22T08:27:06.404664+0000","last_clean":"2024-04-22T08:27:06.404664+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:06.404664+0000","last_undegraded":"2024-04-22T08:27:06.404664+0000","last_fullsized":"2024-04-22T08:27:06.404664+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'4","reported_seq":21,"reported_epoch":28,"state":"active+recovery_unfound","last_fresh":"2024-04-22T08:27:06.410422+0000","last_change":"2024-04-22T08:27:06.410998+0000","last_active":"2024-04-22T08:27:06.410422+0000","last_peered":"2024-04-22T08:27:06.410422+0000","last_clean":"2024-04-22T08:26:38.491250+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:06.410422+0000","last_undegraded":"2024-04-22T08:27:06.380634+0000","last_fullsized":"2024-04-22T08:27:06.410422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:25:59.497393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"19'3","reported_seq":14,"reported_epoch":28,"state":"active+recovery_unfound","last_fresh":"2024-04-22T08:27:06.412509+0000","last_change":"2024-04-22T08:27:06.412836+0000","last_active":"2024-04-22T08:27:06.412509+0000","last_peered":"2024-04-22T08:27:06.412509+0000","last_clean":"2024-04-22T08:26:39.376175+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:06.412509+0000","last_undegraded":"2024-04-22T08:27:06.379523+0000","last_fullsized":"2024-04-22T08:27:06.412509+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:01.624202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.399040+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:06.399040+0000","last_peered":"2024-04-22T08:27:06.399040+0000","last_clean":"2024-04-22T08:27:06.399040+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:06.399040+0000","last_undegraded":"2024-04-22T08:27:06.399040+0000","last_fullsized":"2024-04-22T08:27:06.399040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"19'5","reported_seq":51,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.397710+0000","last_change":"2024-04-22T08:27:06.398148+0000","last_active":"2024-04-22T08:27:06.397710+0000","last_peered":"2024-04-22T08:27:06.397710+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:06.397710+0000","last_undegraded":"2024-04-22T08:27:06.384372+0000","last_fullsized":"2024-04-22T08:27:06.397710+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:25.662296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"19'3","reported_seq":44,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.406062+0000","last_change":"2024-04-22T08:27:06.406509+0000","last_active":"2024-04-22T08:27:06.406062+0000","last_peered":"2024-04-22T08:27:06.406062+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:06.406062+0000","last_undegraded":"2024-04-22T08:27:06.388604+0000","last_fullsized":"2024-04-22T08:27:06.406062+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:05.125941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"19'4","reported_seq":45,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.406133+0000","last_change":"2024-04-22T08:27:06.406551+0000","last_active":"2024-04-22T08:27:06.406133+0000","last_peered":"2024-04-22T08:27:06.406133+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:06.406133+0000","last_undegraded":"2024-04-22T08:27:06.388754+0000","last_fullsized":"2024-04-22T08:27:06.406133+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:51.783707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"19'1","reported_seq":50,"reported_epoch":28,"state":"active+recovering+degraded","last_fresh":"2024-04-22T08:27:06.390696+0000","last_change":"2024-04-22T08:27:06.401985+0000","last_active":"2024-04-22T08:27:06.390696+0000","last_peered":"2024-04-22T08:27:06.390696+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:06.390696+0000","last_undegraded":"2024-04-22T08:27:06.383210+0000","last_fullsized":"2024-04-22T08:27:06.390696+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:20:25.367168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"19'5","reported_seq":14,"reported_epoch":28,"state":"active+recovery_unfound","last_fresh":"2024-04-22T08:27:06.411540+0000","last_change":"2024-04-22T08:27:06.411948+0000","last_active":"2024-04-22T08:27:06.411540+0000","last_peered":"2024-04-22T08:27:06.411540+0000","last_clean":"2024-04-22T08:26:39.458282+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:06.411540+0000","last_undegraded":"2024-04-22T08:27:06.381428+0000","last_fullsized":"2024-04-22T08:27:06.411540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:21.084393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'6","reported_seq":43,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.406217+0000","last_change":"2024-04-22T08:27:06.406525+0000","last_active":"2024-04-22T08:27:06.406217+0000","last_peered":"2024-04-22T08:27:06.406217+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:06.406217+0000","last_undegraded":"2024-04-22T08:27:06.389578+0000","last_fullsized":"2024-04-22T08:27:06.406217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:07:15.622759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"19'3","reported_seq":21,"reported_epoch":28,"state":"active+recovery_unfound","last_fresh":"2024-04-22T08:27:06.402119+0000","last_change":"2024-04-22T08:27:06.402951+0000","last_active":"2024-04-22T08:27:06.402119+0000","last_peered":"2024-04-22T08:27:06.402119+0000","last_clean":"2024-04-22T08:26:38.244414+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:06.402119+0000","last_undegraded":"2024-04-22T08:27:06.379474+0000","last_fullsized":"2024-04-22T08:27:06.402119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:58.882515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"19'4","reported_seq":21,"reported_epoch":28,"state":"active+recovery_unfound","last_fresh":"2024-04-22T08:27:06.408625+0000","last_change":"2024-04-22T08:27:06.409089+0000","last_active":"2024-04-22T08:27:06.408625+0000","last_peered":"2024-04-22T08:27:06.408625+0000","last_clean":"2024-04-22T08:26:38.888837+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:06.408625+0000","last_undegraded":"2024-04-22T08:27:06.379398+0000","last_fullsized":"2024-04-22T08:27:06.408625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:48:11.903425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"19'1","reported_seq":41,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.405921+0000","last_change":"2024-04-22T08:27:06.406342+0000","last_active":"2024-04-22T08:27:06.405921+0000","last_peered":"2024-04-22T08:27:06.405921+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:06.405921+0000","last_undegraded":"2024-04-22T08:27:06.389674+0000","last_fullsized":"2024-04-22T08:27:06.405921+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:34:15.896827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"19'5","reported_seq":46,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.407233+0000","last_change":"2024-04-22T08:27:06.407607+0000","last_active":"2024-04-22T08:27:06.407233+0000","last_peered":"2024-04-22T08:27:06.407233+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:06.407233+0000","last_undegraded":"2024-04-22T08:27:06.390274+0000","last_fullsized":"2024-04-22T08:27:06.407233+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:40.467647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"19'2","reported_seq":42,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.406494+0000","last_change":"2024-04-22T08:27:06.406945+0000","last_active":"2024-04-22T08:27:06.406494+0000","last_peered":"2024-04-22T08:27:06.406494+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:06.406494+0000","last_undegraded":"2024-04-22T08:27:06.390162+0000","last_fullsized":"2024-04-22T08:27:06.406494+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:25:27.628668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"19'9","reported_seq":50,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:06.404324+0000","last_change":"2024-04-22T08:27:06.404802+0000","last_active":"2024-04-22T08:27:06.404324+0000","last_peered":"2024-04-22T08:27:06.404324+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:06.404324+0000","last_undegraded":"2024-04-22T08:27:06.384863+0000","last_fullsized":"2024-04-22T08:27:06.404324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:49.503636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964116995,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":232,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":237568,"data_stored":73444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":32370383},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":270336,"data_stored":77884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":32304851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:11.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:27:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:11.887 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 53/56 objects degraded (94.643%), 9 pgs degraded","count":9},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":28,"num_osds":3,"num_up_osds":2,"osd_up_since":1713774425,"num_in_osds":2,"osd_in_since":1713774425,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+degraded","count":8},{"state_name":"active+recovery_unfound","count":5},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":24,"num_pools":2,"num_objects":28,"data_bytes":17276,"bytes_used":65224704,"bytes_avail":193208303616,"bytes_total":193273528320,"degraded_objects":53,"degraded_total":56,"degraded_ratio":0.9464285714285714,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-22T08:26:05.083154+0000","services":{}},"progress_events":{}} 2024-04-22T08:27:11.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '7ee9519b-73a7-4b76-a33e-63cd80692d01', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 53/56 objects degraded (94.643%), 9 pgs degraded', 'count': 9}, 'muted': False}}, 'mutes': []}, 'election_epoch': 12, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 66, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1713774425, 'num_in_osds': 2, 'osd_in_since': 1713774425, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 10}, {'state_name': 'active+recovery_wait+degraded', 'count': 8}, {'state_name': 'active+recovery_unfound', 'count': 5}, {'state_name': 'active+recovering+degraded', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 28, 'data_bytes': 17276, 'bytes_used': 65224704, 'bytes_avail': 193208303616, 'bytes_total': 193273528320, 'degraded_objects': 53, 'degraded_total': 56, 'degraded_ratio': 0.9464285714285714, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-04-22T08:26:05.083154+0000', 'services': {}}, 'progress_events': {}} 2024-04-22T08:27:11.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-04-22T08:27:11.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:27:11.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:27:12.049 INFO:teuthology.orchestra.run.smithi049.stdout:115964116996 2024-04-22T08:27:12.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:27:12.064 INFO:teuthology.orchestra.run.smithi049.stdout:90194313224 2024-04-22T08:27:12.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:27:12.324 INFO:teuthology.orchestra.run.smithi049.stdout:115964116996 2024-04-22T08:27:12.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 115964116996 got 115964116996 for osd.2 2024-04-22T08:27:12.341 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:12.389 INFO:teuthology.orchestra.run.smithi049.stdout:90194313224 2024-04-22T08:27:12.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313224 got 90194313224 for osd.0 2024-04-22T08:27:12.407 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:12.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:27:12.748 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:12.748 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 53/56 objects degraded (94.643%), 9 pgs degraded","count":9},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":67,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":28,"num_osds":3,"num_up_osds":2,"osd_up_since":1713774425,"num_in_osds":2,"osd_in_since":1713774425,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+degraded","count":8},{"state_name":"active+recovery_unfound","count":5},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":24,"num_pools":2,"num_objects":28,"data_bytes":17276,"bytes_used":65224704,"bytes_avail":193208303616,"bytes_total":193273528320,"degraded_objects":53,"degraded_total":56,"degraded_ratio":0.9464285714285714,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T08:27:12.134296+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:27:12.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '7ee9519b-73a7-4b76-a33e-63cd80692d01', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 53/56 objects degraded (94.643%), 9 pgs degraded', 'count': 9}, 'muted': False}}, 'mutes': []}, 'election_epoch': 12, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 67, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1713774425, 'num_in_osds': 2, 'osd_in_since': 1713774425, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 10}, {'state_name': 'active+recovery_wait+degraded', 'count': 8}, {'state_name': 'active+recovery_unfound', 'count': 5}, {'state_name': 'active+recovering+degraded', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 28, 'data_bytes': 17276, 'bytes_used': 65224704, 'bytes_avail': 193208303616, 'bytes_total': 193273528320, 'degraded_objects': 53, 'degraded_total': 56, 'degraded_ratio': 0.9464285714285714, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-04-22T08:27:12.134296+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-04-22T08:27:12.765 INFO:tasks.rep_lost_unfound_delete:there are 27 unfound objects 2024-04-22T08:27:12.765 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unfounddel_pool -t 20 bench 240 write' 2024-04-22T08:27:12.809 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:hints = 1 2024-04-22T08:27:12.809 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Maintaining 20 concurrent writes of 4096 bytes to objects of size 4096 for up to 240 seconds or 0 objects 2024-04-22T08:27:12.809 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Object prefix: benchmark_data_smithi049_32359 2024-04-22T08:27:12.809 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:27:12.809 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-04-22T08:27:13.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 1 20 2262 2242 8.75283 8.75781 0.00448202 0.00888302 2024-04-22T08:27:14.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 2 20 4617 4597 8.9756 9.19922 0.00803435 0.00869325 2024-04-22T08:27:15.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 3 20 7522 7502 9.76586 11.3477 0.00786406 0.00798229 2024-04-22T08:27:16.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 4 20 10226 10206 9.9648 10.5625 0.00822274 0.00782664 2024-04-22T08:27:17.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 5 20 12816 12796 9.99511 10.1172 0.00969476 0.00780681 2024-04-22T08:27:18.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 6 20 14796 14776 9.61817 7.73438 0.0075983 0.00811323 2024-04-22T08:27:19.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 7 20 17193 17173 9.58165 9.36328 0.012384 0.00814749 2024-04-22T08:27:20.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 8 20 18365 18345 8.95621 4.57812 0.0093283 0.00871517 2024-04-22T08:27:21.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 9 20 20038 20018 8.68715 6.53516 0.00746791 0.00898494 2024-04-22T08:27:22.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:22.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 10 20 22183 22163 8.65612 8.37891 0.012146 0.00901834 2024-04-22T08:27:23.340 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:23.341 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:23.356 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-22T08:27:22.138541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59417468,"num_objects":14530,"num_object_clones":0,"num_object_copies":29060,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":14530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14557,"num_write_kb":58054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14557,"ondisk_log_size":14557,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":201324,"kb_used_data":107804,"kb_used_omap":8,"kb_used_meta":93495,"kb_avail":188542356,"statfs":{"total":193273528320,"available":193067372544,"internally_reserved":0,"allocated":110391296,"data_stored":110034720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":95739298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,16,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":59400192,"num_objects":14502,"num_object_clones":0,"num_object_copies":29004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14502,"num_write_kb":58008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005152"},"pg_stats":[{"pgid":"2.f","version":"28'835","reported_seq":875,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:18.282403+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:18.282403+0000","last_peered":"2024-04-22T08:27:18.282403+0000","last_clean":"2024-04-22T08:27:18.282403+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:18.282403+0000","last_undegraded":"2024-04-22T08:27:18.282403+0000","last_fullsized":"2024-04-22T08:27:18.282403+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3420160,"num_objects":835,"num_object_clones":0,"num_object_copies":1670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":3340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"28'1032","reported_seq":1048,"reported_epoch":28,"state":"active+recovery_unfound+degraded","last_fresh":"2024-04-22T08:27:19.379177+0000","last_change":"2024-04-22T08:27:12.819916+0000","last_active":"2024-04-22T08:27:19.379177+0000","last_peered":"2024-04-22T08:27:19.379177+0000","last_clean":"2024-04-22T08:26:38.491250+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:19.379177+0000","last_undegraded":"2024-04-22T08:27:06.410998+0000","last_fullsized":"2024-04-22T08:27:19.379177+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:25:59.497393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4211922,"num_objects":1030,"num_object_clones":0,"num_object_copies":2060,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1032,"num_write_kb":4116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"28'960","reported_seq":970,"reported_epoch":28,"state":"active+recovery_unfound+degraded","last_fresh":"2024-04-22T08:27:19.381166+0000","last_change":"2024-04-22T08:27:12.820073+0000","last_active":"2024-04-22T08:27:19.381166+0000","last_peered":"2024-04-22T08:27:19.381166+0000","last_clean":"2024-04-22T08:26:39.376175+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:19.381166+0000","last_undegraded":"2024-04-22T08:27:06.412836+0000","last_fullsized":"2024-04-22T08:27:19.381166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:01.624202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3920489,"num_objects":958,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":3830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"28'1034","reported_seq":1054,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:19.366852+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:19.366852+0000","last_peered":"2024-04-22T08:27:19.366852+0000","last_clean":"2024-04-22T08:27:19.366852+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:19.366852+0000","last_undegraded":"2024-04-22T08:27:19.366852+0000","last_fullsized":"2024-04-22T08:27:19.366852+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1034,"log_dups_size":0,"ondisk_log_size":1034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4235264,"num_objects":1034,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"28'865","reported_seq":911,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.289946+0000","last_change":"2024-04-22T08:27:06.398148+0000","last_active":"2024-04-22T08:27:18.289946+0000","last_peered":"2024-04-22T08:27:18.289946+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:18.289946+0000","last_undegraded":"2024-04-22T08:27:06.384372+0000","last_fullsized":"2024-04-22T08:27:18.289946+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:25.662296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3523794,"num_objects":862,"num_object_clones":0,"num_object_copies":1724,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":865,"num_write_kb":3444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":862}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"28'860","reported_seq":901,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.269549+0000","last_change":"2024-04-22T08:27:06.406509+0000","last_active":"2024-04-22T08:27:18.269549+0000","last_peered":"2024-04-22T08:27:18.269549+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:18.269549+0000","last_undegraded":"2024-04-22T08:27:06.388604+0000","last_fullsized":"2024-04-22T08:27:18.269549+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:05.125941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3511506,"num_objects":859,"num_object_clones":0,"num_object_copies":1718,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":860,"num_write_kb":3431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"28'863","reported_seq":904,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.283972+0000","last_change":"2024-04-22T08:27:06.406551+0000","last_active":"2024-04-22T08:27:18.283972+0000","last_peered":"2024-04-22T08:27:18.283972+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:18.283972+0000","last_undegraded":"2024-04-22T08:27:06.388754+0000","last_fullsized":"2024-04-22T08:27:18.283972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":863,"log_dups_size":0,"ondisk_log_size":863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:51.783707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3519698,"num_objects":861,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":861}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"28'837","reported_seq":886,"reported_epoch":28,"state":"active+recovering","last_fresh":"2024-04-22T08:27:18.288521+0000","last_change":"2024-04-22T08:27:12.823560+0000","last_active":"2024-04-22T08:27:18.288521+0000","last_peered":"2024-04-22T08:27:18.288521+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:18.288521+0000","last_undegraded":"2024-04-22T08:27:18.288521+0000","last_fullsized":"2024-04-22T08:27:18.288521+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:20:25.367168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3424873,"num_objects":837,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":3345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":837}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"28'1049","reported_seq":1058,"reported_epoch":28,"state":"active+recovery_unfound+degraded","last_fresh":"2024-04-22T08:27:19.372904+0000","last_change":"2024-04-22T08:27:12.820771+0000","last_active":"2024-04-22T08:27:19.372904+0000","last_peered":"2024-04-22T08:27:19.372904+0000","last_clean":"2024-04-22T08:26:39.458282+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:19.372904+0000","last_undegraded":"2024-04-22T08:27:06.411948+0000","last_fullsized":"2024-04-22T08:27:19.372904+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:21.084393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4277458,"num_objects":1046,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1049,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"28'817","reported_seq":854,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.284255+0000","last_change":"2024-04-22T08:27:06.406525+0000","last_active":"2024-04-22T08:27:18.284255+0000","last_peered":"2024-04-22T08:27:18.284255+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:18.284255+0000","last_undegraded":"2024-04-22T08:27:06.389578+0000","last_fullsized":"2024-04-22T08:27:18.284255+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:07:15.622759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3323090,"num_objects":813,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":817,"num_write_kb":3248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"28'974","reported_seq":992,"reported_epoch":28,"state":"active+recovery_unfound+degraded","last_fresh":"2024-04-22T08:27:19.357042+0000","last_change":"2024-04-22T08:27:12.820954+0000","last_active":"2024-04-22T08:27:19.357042+0000","last_peered":"2024-04-22T08:27:19.357042+0000","last_clean":"2024-04-22T08:26:38.244414+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:19.357042+0000","last_undegraded":"2024-04-22T08:27:06.402951+0000","last_fullsized":"2024-04-22T08:27:19.357042+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:58.882515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978450,"num_objects":973,"num_object_clones":0,"num_object_copies":1946,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":974,"num_write_kb":3887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":973}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"28'1007","reported_seq":1024,"reported_epoch":28,"state":"active+recovery_unfound+degraded","last_fresh":"2024-04-22T08:27:19.368010+0000","last_change":"2024-04-22T08:27:12.819518+0000","last_active":"2024-04-22T08:27:19.368010+0000","last_peered":"2024-04-22T08:27:19.368010+0000","last_clean":"2024-04-22T08:26:38.888837+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:19.368010+0000","last_undegraded":"2024-04-22T08:27:06.409089+0000","last_fullsized":"2024-04-22T08:27:19.368010+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:48:11.903425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4109522,"num_objects":1005,"num_object_clones":0,"num_object_copies":2010,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1007,"num_write_kb":4015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1005}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"28'814","reported_seq":854,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.282905+0000","last_change":"2024-04-22T08:27:06.406342+0000","last_active":"2024-04-22T08:27:18.282905+0000","last_peered":"2024-04-22T08:27:18.282905+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:18.282905+0000","last_undegraded":"2024-04-22T08:27:06.389674+0000","last_fullsized":"2024-04-22T08:27:18.282905+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:34:15.896827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3330665,"num_objects":814,"num_object_clones":0,"num_object_copies":1628,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":814,"num_write_kb":3253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"28'881","reported_seq":922,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.269485+0000","last_change":"2024-04-22T08:27:06.407607+0000","last_active":"2024-04-22T08:27:18.269485+0000","last_peered":"2024-04-22T08:27:18.269485+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:18.269485+0000","last_undegraded":"2024-04-22T08:27:06.390274+0000","last_fullsized":"2024-04-22T08:27:18.269485+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:40.467647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3589947,"num_objects":879,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":3508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"28'856","reported_seq":895,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.285649+0000","last_change":"2024-04-22T08:27:06.406945+0000","last_active":"2024-04-22T08:27:18.285649+0000","last_peered":"2024-04-22T08:27:18.285649+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:18.285649+0000","last_undegraded":"2024-04-22T08:27:06.390162+0000","last_fullsized":"2024-04-22T08:27:18.285649+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":856,"log_dups_size":0,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:25:27.628668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3499218,"num_objects":856,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":3418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'873","reported_seq":914,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:18.278000+0000","last_change":"2024-04-22T08:27:06.404802+0000","last_active":"2024-04-22T08:27:18.278000+0000","last_peered":"2024-04-22T08:27:18.278000+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:18.278000+0000","last_undegraded":"2024-04-22T08:27:06.384863+0000","last_fullsized":"2024-04-22T08:27:18.278000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:49.503636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3541412,"num_objects":868,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":873,"num_write_kb":3463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":59417468,"num_objects":14530,"num_object_clones":0,"num_object_copies":29060,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":14530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14557,"num_write_kb":58054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":121647104,"data_stored":121514902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14557,"ondisk_log_size":14557,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964116998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117604,"kb_used_data":59740,"kb_used_omap":4,"kb_used_meta":57851,"kb_avail":94254236,"statfs":{"total":96636764160,"available":96516337664,"internally_reserved":0,"allocated":61173760,"data_stored":61009636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":59240143},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2999999999999998}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313226,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83720,"kb_used_data":48064,"kb_used_omap":4,"kb_used_meta":35643,"kb_avail":94288120,"statfs":{"total":96636764160,"available":96551034880,"internally_reserved":0,"allocated":49217536,"data_stored":49025084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":36499155},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,10,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55345152,"data_stored":55310362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66187264,"data_stored":66187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:23.358 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.f 2024-04-22T08:27:23.358 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.e state active+recovery_unfound+degraded 2024-04-22T08:27:23.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.e list_unfound '{}' 2024-04-22T08:27:23.598 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:23.599 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_4", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1224592654, 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'1", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:23.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_6", 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3101443614, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'2", 2024-04-22T08:27:23.601 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "NotRecovering", 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:23.602 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:23.603 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:23.624 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.e on osd.2 2024-04-22T08:27:23.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.e mark_unfound_lost delete 2024-04-22T08:27:23.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 11 20 23656 23636 8.39226 5.75391 0.00804856 0.00930057 2024-04-22T08:27:23.918 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:23.990 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.d state active+recovery_unfound+degraded 2024-04-22T08:27:23.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.d list_unfound '{}' 2024-04-22T08:27:24.228 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 1, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 1, 2024-04-22T08:27:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_7", 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 2358579165, 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "NotRecovering", 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:24.231 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:24.232 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:24.232 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:24.277 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.d on osd.2 2024-04-22T08:27:24.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.d mark_unfound_lost delete 2024-04-22T08:27:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-22T08:27:24.558 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.c 2024-04-22T08:27:24.559 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.b state active+recovery_wait+degraded 2024-04-22T08:27:24.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.b list_unfound '{}' 2024-04-22T08:27:24.733 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_3", 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 943960171, 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'5", 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_1", 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 2885884443, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'1", 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:24.738 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:24.747 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.b on osd.0 2024-04-22T08:27:24.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.b mark_unfound_lost delete 2024-04-22T08:27:24.810 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 12 20 25249 25229 8.21143 6.22266 0.0191985 0.00941528 2024-04-22T08:27:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:24.937 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.a state active+recovery_wait+degraded 2024-04-22T08:27:24.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.a list_unfound '{}' 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:25.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_9", 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:25.194 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1217469130, 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'1", 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_9", 2024-04-22T08:27:25.195 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 170713514, 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'2", 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:25.196 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:25.197 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:25.223 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.a on osd.0 2024-04-22T08:27:25.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.a mark_unfound_lost delete 2024-04-22T08:27:25.505 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:25.535 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.9 state active+recovery_wait+degraded 2024-04-22T08:27:25.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.9 list_unfound '{}' 2024-04-22T08:27:25.811 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 13 20 26880 26860 8.06983 6.37109 0.0145099 0.0096734 2024-04-22T08:27:25.823 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_5", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 2998549577, 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'2", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_2", 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 4032093161, 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'1", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:25.828 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:25.856 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.9 on osd.0 2024-04-22T08:27:25.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.9 mark_unfound_lost delete 2024-04-22T08:27:26.150 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:26.189 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.8 state active+recovering 2024-04-22T08:27:26.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.8 list_unfound '{}' 2024-04-22T08:27:26.317 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:27:26.316+0000 7f4034a32640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 4 pgs recovery_unfound (PG_DAMAGED) 2024-04-22T08:27:26.556 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 1, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 1, 2024-04-22T08:27:26.557 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_8", 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 2730327960, 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'1", 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:26.558 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:26.559 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:26.589 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.8 on osd.0 2024-04-22T08:27:26.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.8 mark_unfound_lost delete 2024-04-22T08:27:26.811 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 14 20 28627 28607 7.98083 6.82422 0.0132098 0.00977982 2024-04-22T08:27:26.918 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-22T08:27:26.949 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.0 2024-04-22T08:27:26.950 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.3 state active+recovery_unfound+degraded 2024-04-22T08:27:26.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.3 list_unfound '{}' 2024-04-22T08:27:27.252 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:27.252 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:27.252 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:27.252 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.252 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_8", 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:27.253 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1901787523, 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'5", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'3", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_1", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.254 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3122379363, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "NotRecovering", 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:27.256 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:27.256 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:27.256 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:27.256 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:27.256 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:27.278 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.3 on osd.2 2024-04-22T08:27:27.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.3 mark_unfound_lost delete 2024-04-22T08:27:27.610 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:27.653 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.1 2024-04-22T08:27:27.653 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.2 state active+recovery_wait+degraded 2024-04-22T08:27:27.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.2 list_unfound '{}' 2024-04-22T08:27:27.811 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 15 19 30308 30289 7.88671 6.57031 0.020481 0.00989733 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_6", 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1708673474, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'5", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_9", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 831402034, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'6", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:27.892 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:27.918 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.2 on osd.0 2024-04-22T08:27:27.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.2 mark_unfound_lost delete 2024-04-22T08:27:28.176 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:28.209 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.2 2024-04-22T08:27:28.209 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.1 state active+recovery_unfound+degraded 2024-04-22T08:27:28.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.1 list_unfound '{}' 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_3", 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 459768081, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'1", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_3", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 480221105, 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'2", 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "NotRecovering", 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:28.459 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:28.481 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.1 on osd.2 2024-04-22T08:27:28.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.1 mark_unfound_lost delete 2024-04-22T08:27:28.721 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:28.768 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.3 2024-04-22T08:27:28.768 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.0 state active+recovery_unfound+degraded 2024-04-22T08:27:28.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2024-04-22T08:27:28.811 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 16 20 31904 31884 7.78317 6.23047 0.014064 0.0100298 2024-04-22T08:27:29.022 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_5", 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 4248945696, 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_5", 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 4212789968, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "NotRecovering", 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:29.027 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:29.084 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.0 on osd.2 2024-04-22T08:27:29.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2024-04-22T08:27:29.497 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:29.533 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.7 state active+recovery_wait+degraded 2024-04-22T08:27:29.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.7 list_unfound '{}' 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 1, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 1, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_7", 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3761017527, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'1", 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:29.812 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 17 20 33495 33475 7.69071 6.21484 0.0169359 0.0101518 2024-04-22T08:27:29.837 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.7 on osd.0 2024-04-22T08:27:29.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.7 mark_unfound_lost delete 2024-04-22T08:27:30.103 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-22T08:27:30.155 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.4 2024-04-22T08:27:30.155 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.6 state active+recovery_wait+degraded 2024-04-22T08:27:30.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.6 list_unfound '{}' 2024-04-22T08:27:30.522 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:30.522 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:30.522 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 3, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 3, 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_8", 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:30.523 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1861424966, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'5", 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_6", 2024-04-22T08:27:30.524 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3910131350, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'4", 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:30.525 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_4", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 2787839990, 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'3", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:30.526 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:30.527 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:30.580 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.6 on osd.0 2024-04-22T08:27:30.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.6 mark_unfound_lost delete 2024-04-22T08:27:30.812 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 18 19 35085 35066 7.6087 6.21484 0.0167431 0.010261 2024-04-22T08:27:30.949 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 3 objects unfound and apparently lost marking 2024-04-22T08:27:30.978 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.5 2024-04-22T08:27:30.979 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.5 state active+recovery_wait+degraded 2024-04-22T08:27:30.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.5 list_unfound '{}' 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 2, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 2, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_1", 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 26281365, 2024-04-22T08:27:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'1", 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "new_2", 2024-04-22T08:27:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3196340981, 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'2", 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:31.228 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:31.249 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.5 on osd.0 2024-04-22T08:27:31.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.5 mark_unfound_lost delete 2024-04-22T08:27:31.574 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-22T08:27:31.605 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.6 2024-04-22T08:27:31.605 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.7 2024-04-22T08:27:31.605 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.4 state active+recovery_wait+degraded 2024-04-22T08:27:31.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.4 list_unfound '{}' 2024-04-22T08:27:31.812 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 19 20 36605 36585 7.52051 5.93359 0.0109029 0.0103786 2024-04-22T08:27:31.851 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "offset": { 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "", 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": 0, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 0, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": -9223372036854775808, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "num_missing": 4, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "num_unfound": 3, 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: "objects": [ 2024-04-22T08:27:31.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_4", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 873891716, 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'8", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:31.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existing_7", 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 3996155460, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'9", 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "17'6", 2024-04-22T08:27:31.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": { 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "oid": "existed_2", 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "snapid": -2, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "hash": 1767667508, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 0, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "namespace": "" 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-22T08:27:31.855 INFO:teuthology.orchestra.run.smithi049.stdout: "need": "19'7", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "have": "0'0", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "none", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "locations": [] 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "Active", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "available_might_have_unfound": true, 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "might_have_unfound": [ 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": "1", 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "osd is down" 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-22T08:27:31.856 INFO:teuthology.orchestra.run.smithi049.stdout: "more": false 2024-04-22T08:27:31.857 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:31.901 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.4 on osd.0 2024-04-22T08:27:31.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.4 mark_unfound_lost delete 2024-04-22T08:27:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:pg has 3 objects unfound and apparently lost marking 2024-04-22T08:27:32.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-04-22T08:27:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-04-22T08:27:32.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-04-22T08:27:32.812 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:27:32.816991+0000 min lat: 0.00109492 max lat: 0.229596 avg lat: 0.010488 2024-04-22T08:27:32.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:27:32.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 20 20 38135 38115 7.44329 5.97656 0.0049426 0.010488 2024-04-22T08:27:33.007 INFO:teuthology.orchestra.run.smithi049.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-04-22T08:27:33.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:27:33.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:27:33.396 INFO:teuthology.orchestra.run.smithi049.stdout:115964117001 2024-04-22T08:27:33.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:27:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:90194313230 2024-04-22T08:27:33.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:27:33.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 21 20 39576 39556 7.35687 5.62891 0.0115485 0.010611 2024-04-22T08:27:33.987 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:27:33.987+0000 7f4034a32640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2024-04-22T08:27:34.084 INFO:teuthology.orchestra.run.smithi049.stdout:115964117000 2024-04-22T08:27:34.099 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 115964117001 got 115964117000 for osd.2 2024-04-22T08:27:34.116 INFO:teuthology.orchestra.run.smithi049.stdout:90194313228 2024-04-22T08:27:34.131 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313230 got 90194313228 for osd.0 2024-04-22T08:27:34.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 22 20 40788 40768 7.23766 4.73438 0.0133527 0.0107869 2024-04-22T08:27:35.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:27:35.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:27:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:115964117001 2024-04-22T08:27:35.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 115964117001 got 115964117001 for osd.2 2024-04-22T08:27:35.759 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:35.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 23 20 42168 42148 7.15734 5.39062 0.0146195 0.0109093 2024-04-22T08:27:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:90194313230 2024-04-22T08:27:35.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 90194313230 got 90194313230 for osd.0 2024-04-22T08:27:35.957 DEBUG:teuthology.parallel:result is None 2024-04-22T08:27:35.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-04-22T08:27:35.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:36.421 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:36.423 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:36.463 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-22T08:27:36.144848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":530060,"kb_used_data":310200,"kb_used_omap":8,"kb_used_meta":219831,"kb_avail":188213620,"statfs":{"total":193273528320,"available":192730746880,"internally_reserved":0,"allocated":317644800,"data_stored":317302140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":225107362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,11,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":77783040,"num_objects":18990,"num_object_clones":0,"num_object_copies":37980,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":18990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18990,"num_write_kb":75960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005942"},"pg_stats":[{"pgid":"2.f","version":"28'2428","reported_seq":2468,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:33.369967+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:33.369967+0000","last_peered":"2024-04-22T08:27:33.369967+0000","last_clean":"2024-04-22T08:27:33.369967+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:33.369967+0000","last_undegraded":"2024-04-22T08:27:33.369967+0000","last_fullsized":"2024-04-22T08:27:33.369967+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":0,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":2428,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"28'2512","reported_seq":2540,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.400005+0000","last_change":"2024-04-22T08:27:23.976123+0000","last_active":"2024-04-22T08:27:34.400005+0000","last_peered":"2024-04-22T08:27:34.400005+0000","last_clean":"2024-04-22T08:27:34.400005+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:34.400005+0000","last_undegraded":"2024-04-22T08:27:34.400005+0000","last_fullsized":"2024-04-22T08:27:34.400005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10265810,"num_objects":2508,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2510,"num_write_kb":10028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"28'2451","reported_seq":2470,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.399953+0000","last_change":"2024-04-22T08:27:24.544733+0000","last_active":"2024-04-22T08:27:34.399953+0000","last_peered":"2024-04-22T08:27:34.399953+0000","last_clean":"2024-04-22T08:27:34.399953+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:34.399953+0000","last_undegraded":"2024-04-22T08:27:34.399953+0000","last_fullsized":"2024-04-22T08:27:34.399953+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2451,"log_dups_size":0,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10023529,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"28'2574","reported_seq":2594,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.389546+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:34.389546+0000","last_peered":"2024-04-22T08:27:34.389546+0000","last_clean":"2024-04-22T08:27:34.389546+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:34.389546+0000","last_undegraded":"2024-04-22T08:27:34.389546+0000","last_fullsized":"2024-04-22T08:27:34.389546+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":10296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"28'2444","reported_seq":2490,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.352060+0000","last_change":"2024-04-22T08:27:06.398148+0000","last_active":"2024-04-22T08:27:33.352060+0000","last_peered":"2024-04-22T08:27:33.352060+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:33.352060+0000","last_undegraded":"2024-04-22T08:27:06.384372+0000","last_fullsized":"2024-04-22T08:27:33.352060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983186,"num_objects":2439,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":9752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"28'2506","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.384104+0000","last_change":"2024-04-22T08:27:06.406509+0000","last_active":"2024-04-22T08:27:33.384104+0000","last_peered":"2024-04-22T08:27:33.384104+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:33.384104+0000","last_undegraded":"2024-04-22T08:27:06.388604+0000","last_fullsized":"2024-04-22T08:27:33.384104+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245330,"num_objects":2503,"num_object_clones":0,"num_object_copies":5006,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2503}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"28'2411","reported_seq":2452,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.359463+0000","last_change":"2024-04-22T08:27:06.406551+0000","last_active":"2024-04-22T08:27:33.359463+0000","last_peered":"2024-04-22T08:27:33.359463+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:33.359463+0000","last_undegraded":"2024-04-22T08:27:06.388754+0000","last_fullsized":"2024-04-22T08:27:33.359463+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9852114,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":9624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"28'2439","reported_seq":2489,"reported_epoch":28,"state":"active+recovering","last_fresh":"2024-04-22T08:27:33.371304+0000","last_change":"2024-04-22T08:27:12.823560+0000","last_active":"2024-04-22T08:27:33.371304+0000","last_peered":"2024-04-22T08:27:33.371304+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:33.371304+0000","last_undegraded":"2024-04-22T08:27:33.371304+0000","last_fullsized":"2024-04-22T08:27:33.371304+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9982569,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":9749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"28'2581","reported_seq":2600,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.408164+0000","last_change":"2024-04-22T08:27:27.644057+0000","last_active":"2024-04-22T08:27:34.408164+0000","last_peered":"2024-04-22T08:27:34.408164+0000","last_clean":"2024-04-22T08:27:34.408164+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:34.408164+0000","last_undegraded":"2024-04-22T08:27:34.408164+0000","last_fullsized":"2024-04-22T08:27:34.408164+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10544338,"num_objects":2576,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"28'2422","reported_seq":2459,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.344361+0000","last_change":"2024-04-22T08:27:06.406525+0000","last_active":"2024-04-22T08:27:33.344361+0000","last_peered":"2024-04-22T08:27:33.344361+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:33.344361+0000","last_undegraded":"2024-04-22T08:27:06.389578+0000","last_fullsized":"2024-04-22T08:27:33.344361+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2422,"log_dups_size":0,"ondisk_log_size":2422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9888978,"num_objects":2416,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":9660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2416}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"28'2452","reported_seq":2481,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.324110+0000","last_change":"2024-04-22T08:27:28.775685+0000","last_active":"2024-04-22T08:27:34.324110+0000","last_peered":"2024-04-22T08:27:34.324110+0000","last_clean":"2024-04-22T08:27:34.324110+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:34.324110+0000","last_undegraded":"2024-04-22T08:27:34.324110+0000","last_fullsized":"2024-04-22T08:27:34.324110+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024146,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"28'2528","reported_seq":2556,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.375833+0000","last_change":"2024-04-22T08:27:29.540465+0000","last_active":"2024-04-22T08:27:34.375833+0000","last_peered":"2024-04-22T08:27:34.375833+0000","last_clean":"2024-04-22T08:27:34.375833+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:34.375833+0000","last_undegraded":"2024-04-22T08:27:34.375833+0000","last_fullsized":"2024-04-22T08:27:34.375833+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2528,"log_dups_size":0,"ondisk_log_size":2528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10331346,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"28'2368","reported_seq":2408,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.370021+0000","last_change":"2024-04-22T08:27:06.406342+0000","last_active":"2024-04-22T08:27:33.370021+0000","last_peered":"2024-04-22T08:27:33.370021+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:33.370021+0000","last_undegraded":"2024-04-22T08:27:06.389674+0000","last_fullsized":"2024-04-22T08:27:33.370021+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2368,"log_dups_size":0,"ondisk_log_size":2368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9691753,"num_objects":2367,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2367,"num_write_kb":9465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2367}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"28'2426","reported_seq":2465,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.361596+0000","last_change":"2024-04-22T08:27:06.407607+0000","last_active":"2024-04-22T08:27:33.361596+0000","last_peered":"2024-04-22T08:27:33.361596+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:33.361596+0000","last_undegraded":"2024-04-22T08:27:06.390274+0000","last_fullsized":"2024-04-22T08:27:33.361596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2426,"log_dups_size":0,"ondisk_log_size":2426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9905979,"num_objects":2421,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2423,"num_write_kb":9676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2421}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"28'2395","reported_seq":2431,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.378805+0000","last_change":"2024-04-22T08:27:06.406945+0000","last_active":"2024-04-22T08:27:33.378805+0000","last_peered":"2024-04-22T08:27:33.378805+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:33.378805+0000","last_undegraded":"2024-04-22T08:27:06.390162+0000","last_fullsized":"2024-04-22T08:27:33.378805+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794770,"num_objects":2393,"num_object_clones":0,"num_object_copies":4786,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":9566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2393}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'2507","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.385566+0000","last_change":"2024-04-22T08:27:06.404802+0000","last_active":"2024-04-22T08:27:33.385566+0000","last_peered":"2024-04-22T08:27:33.385566+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:33.385566+0000","last_undegraded":"2024-04-22T08:27:06.384863+0000","last_fullsized":"2024-04-22T08:27:33.385566+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10221988,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":9987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2498}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324100096,"data_stored":323981810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270992,"kb_used_data":158980,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94100848,"statfs":{"total":96636764160,"available":96359268352,"internally_reserved":0,"allocated":162795520,"data_stored":162631396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114683599},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19500000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259068,"kb_used_data":151220,"kb_used_omap":4,"kb_used_meta":107835,"kb_avail":94112772,"statfs":{"total":96636764160,"available":96371478528,"internally_reserved":0,"allocated":154849280,"data_stored":154670744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":110423763},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.18099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159551488,"data_stored":159530614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164433920,"data_stored":164433920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:36.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:36.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 24 20 43425 43405 7.06369 4.91016 0.0134897 0.0110533 2024-04-22T08:27:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:37.142 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:37.173 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-22T08:27:36.144848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":530060,"kb_used_data":310200,"kb_used_omap":8,"kb_used_meta":219831,"kb_avail":188213620,"statfs":{"total":193273528320,"available":192730746880,"internally_reserved":0,"allocated":317644800,"data_stored":317302140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":225107362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,11,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":77783040,"num_objects":18990,"num_object_clones":0,"num_object_copies":37980,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":18990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18990,"num_write_kb":75960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005942"},"pg_stats":[{"pgid":"2.f","version":"28'2428","reported_seq":2468,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:33.369967+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:33.369967+0000","last_peered":"2024-04-22T08:27:33.369967+0000","last_clean":"2024-04-22T08:27:33.369967+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:33.369967+0000","last_undegraded":"2024-04-22T08:27:33.369967+0000","last_fullsized":"2024-04-22T08:27:33.369967+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":0,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":2428,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"28'2512","reported_seq":2540,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.400005+0000","last_change":"2024-04-22T08:27:23.976123+0000","last_active":"2024-04-22T08:27:34.400005+0000","last_peered":"2024-04-22T08:27:34.400005+0000","last_clean":"2024-04-22T08:27:34.400005+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:34.400005+0000","last_undegraded":"2024-04-22T08:27:34.400005+0000","last_fullsized":"2024-04-22T08:27:34.400005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10265810,"num_objects":2508,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2510,"num_write_kb":10028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"28'2451","reported_seq":2470,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.399953+0000","last_change":"2024-04-22T08:27:24.544733+0000","last_active":"2024-04-22T08:27:34.399953+0000","last_peered":"2024-04-22T08:27:34.399953+0000","last_clean":"2024-04-22T08:27:34.399953+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:34.399953+0000","last_undegraded":"2024-04-22T08:27:34.399953+0000","last_fullsized":"2024-04-22T08:27:34.399953+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2451,"log_dups_size":0,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10023529,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"28'2574","reported_seq":2594,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.389546+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:34.389546+0000","last_peered":"2024-04-22T08:27:34.389546+0000","last_clean":"2024-04-22T08:27:34.389546+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:34.389546+0000","last_undegraded":"2024-04-22T08:27:34.389546+0000","last_fullsized":"2024-04-22T08:27:34.389546+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":10296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"28'2444","reported_seq":2490,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.352060+0000","last_change":"2024-04-22T08:27:06.398148+0000","last_active":"2024-04-22T08:27:33.352060+0000","last_peered":"2024-04-22T08:27:33.352060+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:33.352060+0000","last_undegraded":"2024-04-22T08:27:06.384372+0000","last_fullsized":"2024-04-22T08:27:33.352060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983186,"num_objects":2439,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":9752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"28'2506","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.384104+0000","last_change":"2024-04-22T08:27:06.406509+0000","last_active":"2024-04-22T08:27:33.384104+0000","last_peered":"2024-04-22T08:27:33.384104+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:33.384104+0000","last_undegraded":"2024-04-22T08:27:06.388604+0000","last_fullsized":"2024-04-22T08:27:33.384104+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245330,"num_objects":2503,"num_object_clones":0,"num_object_copies":5006,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2503}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"28'2411","reported_seq":2452,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.359463+0000","last_change":"2024-04-22T08:27:06.406551+0000","last_active":"2024-04-22T08:27:33.359463+0000","last_peered":"2024-04-22T08:27:33.359463+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:33.359463+0000","last_undegraded":"2024-04-22T08:27:06.388754+0000","last_fullsized":"2024-04-22T08:27:33.359463+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9852114,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":9624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"28'2439","reported_seq":2489,"reported_epoch":28,"state":"active+recovering","last_fresh":"2024-04-22T08:27:33.371304+0000","last_change":"2024-04-22T08:27:12.823560+0000","last_active":"2024-04-22T08:27:33.371304+0000","last_peered":"2024-04-22T08:27:33.371304+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:33.371304+0000","last_undegraded":"2024-04-22T08:27:33.371304+0000","last_fullsized":"2024-04-22T08:27:33.371304+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9982569,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":9749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"28'2581","reported_seq":2600,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.408164+0000","last_change":"2024-04-22T08:27:27.644057+0000","last_active":"2024-04-22T08:27:34.408164+0000","last_peered":"2024-04-22T08:27:34.408164+0000","last_clean":"2024-04-22T08:27:34.408164+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:34.408164+0000","last_undegraded":"2024-04-22T08:27:34.408164+0000","last_fullsized":"2024-04-22T08:27:34.408164+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10544338,"num_objects":2576,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"28'2422","reported_seq":2459,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.344361+0000","last_change":"2024-04-22T08:27:06.406525+0000","last_active":"2024-04-22T08:27:33.344361+0000","last_peered":"2024-04-22T08:27:33.344361+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:33.344361+0000","last_undegraded":"2024-04-22T08:27:06.389578+0000","last_fullsized":"2024-04-22T08:27:33.344361+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2422,"log_dups_size":0,"ondisk_log_size":2422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9888978,"num_objects":2416,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":9660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2416}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"28'2452","reported_seq":2481,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.324110+0000","last_change":"2024-04-22T08:27:28.775685+0000","last_active":"2024-04-22T08:27:34.324110+0000","last_peered":"2024-04-22T08:27:34.324110+0000","last_clean":"2024-04-22T08:27:34.324110+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:34.324110+0000","last_undegraded":"2024-04-22T08:27:34.324110+0000","last_fullsized":"2024-04-22T08:27:34.324110+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024146,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"28'2528","reported_seq":2556,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.375833+0000","last_change":"2024-04-22T08:27:29.540465+0000","last_active":"2024-04-22T08:27:34.375833+0000","last_peered":"2024-04-22T08:27:34.375833+0000","last_clean":"2024-04-22T08:27:34.375833+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:34.375833+0000","last_undegraded":"2024-04-22T08:27:34.375833+0000","last_fullsized":"2024-04-22T08:27:34.375833+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2528,"log_dups_size":0,"ondisk_log_size":2528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10331346,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"28'2368","reported_seq":2408,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.370021+0000","last_change":"2024-04-22T08:27:06.406342+0000","last_active":"2024-04-22T08:27:33.370021+0000","last_peered":"2024-04-22T08:27:33.370021+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:33.370021+0000","last_undegraded":"2024-04-22T08:27:06.389674+0000","last_fullsized":"2024-04-22T08:27:33.370021+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2368,"log_dups_size":0,"ondisk_log_size":2368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9691753,"num_objects":2367,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2367,"num_write_kb":9465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2367}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"28'2426","reported_seq":2465,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.361596+0000","last_change":"2024-04-22T08:27:06.407607+0000","last_active":"2024-04-22T08:27:33.361596+0000","last_peered":"2024-04-22T08:27:33.361596+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:33.361596+0000","last_undegraded":"2024-04-22T08:27:06.390274+0000","last_fullsized":"2024-04-22T08:27:33.361596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2426,"log_dups_size":0,"ondisk_log_size":2426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9905979,"num_objects":2421,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2423,"num_write_kb":9676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2421}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"28'2395","reported_seq":2431,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.378805+0000","last_change":"2024-04-22T08:27:06.406945+0000","last_active":"2024-04-22T08:27:33.378805+0000","last_peered":"2024-04-22T08:27:33.378805+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:33.378805+0000","last_undegraded":"2024-04-22T08:27:06.390162+0000","last_fullsized":"2024-04-22T08:27:33.378805+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794770,"num_objects":2393,"num_object_clones":0,"num_object_copies":4786,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":9566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2393}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'2507","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.385566+0000","last_change":"2024-04-22T08:27:06.404802+0000","last_active":"2024-04-22T08:27:33.385566+0000","last_peered":"2024-04-22T08:27:33.385566+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:33.385566+0000","last_undegraded":"2024-04-22T08:27:06.384863+0000","last_fullsized":"2024-04-22T08:27:33.385566+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10221988,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":9987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2498}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324100096,"data_stored":323981810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270992,"kb_used_data":158980,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94100848,"statfs":{"total":96636764160,"available":96359268352,"internally_reserved":0,"allocated":162795520,"data_stored":162631396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114683599},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19500000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259068,"kb_used_data":151220,"kb_used_omap":4,"kb_used_meta":107835,"kb_avail":94112772,"statfs":{"total":96636764160,"available":96371478528,"internally_reserved":0,"allocated":154849280,"data_stored":154670744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":110423763},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.18099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159551488,"data_stored":159530614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164433920,"data_stored":164433920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:37.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:37.777 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:37.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 25 20 44779 44759 6.99269 5.28906 0.024155 0.0111649 2024-04-22T08:27:37.846 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-22T08:27:36.144848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":530060,"kb_used_data":310200,"kb_used_omap":8,"kb_used_meta":219831,"kb_avail":188213620,"statfs":{"total":193273528320,"available":192730746880,"internally_reserved":0,"allocated":317644800,"data_stored":317302140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":225107362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,6,11,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":77783040,"num_objects":18990,"num_object_clones":0,"num_object_copies":37980,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":18990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18990,"num_write_kb":75960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005942"},"pg_stats":[{"pgid":"2.f","version":"28'2428","reported_seq":2468,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:33.369967+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:33.369967+0000","last_peered":"2024-04-22T08:27:33.369967+0000","last_clean":"2024-04-22T08:27:33.369967+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:33.369967+0000","last_undegraded":"2024-04-22T08:27:33.369967+0000","last_fullsized":"2024-04-22T08:27:33.369967+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":0,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":2428,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"28'2512","reported_seq":2540,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.400005+0000","last_change":"2024-04-22T08:27:23.976123+0000","last_active":"2024-04-22T08:27:34.400005+0000","last_peered":"2024-04-22T08:27:34.400005+0000","last_clean":"2024-04-22T08:27:34.400005+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:34.400005+0000","last_undegraded":"2024-04-22T08:27:34.400005+0000","last_fullsized":"2024-04-22T08:27:34.400005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10265810,"num_objects":2508,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2510,"num_write_kb":10028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"28'2451","reported_seq":2470,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.399953+0000","last_change":"2024-04-22T08:27:24.544733+0000","last_active":"2024-04-22T08:27:34.399953+0000","last_peered":"2024-04-22T08:27:34.399953+0000","last_clean":"2024-04-22T08:27:34.399953+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:34.399953+0000","last_undegraded":"2024-04-22T08:27:34.399953+0000","last_fullsized":"2024-04-22T08:27:34.399953+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2451,"log_dups_size":0,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10023529,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"28'2574","reported_seq":2594,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.389546+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:34.389546+0000","last_peered":"2024-04-22T08:27:34.389546+0000","last_clean":"2024-04-22T08:27:34.389546+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:34.389546+0000","last_undegraded":"2024-04-22T08:27:34.389546+0000","last_fullsized":"2024-04-22T08:27:34.389546+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":10296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"28'2444","reported_seq":2490,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.352060+0000","last_change":"2024-04-22T08:27:06.398148+0000","last_active":"2024-04-22T08:27:33.352060+0000","last_peered":"2024-04-22T08:27:33.352060+0000","last_clean":"2024-04-22T08:26:38.330797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:33.352060+0000","last_undegraded":"2024-04-22T08:27:06.384372+0000","last_fullsized":"2024-04-22T08:27:33.352060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983186,"num_objects":2439,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":9752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"28'2506","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.384104+0000","last_change":"2024-04-22T08:27:06.406509+0000","last_active":"2024-04-22T08:27:33.384104+0000","last_peered":"2024-04-22T08:27:33.384104+0000","last_clean":"2024-04-22T08:26:39.701927+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:33.384104+0000","last_undegraded":"2024-04-22T08:27:06.388604+0000","last_fullsized":"2024-04-22T08:27:33.384104+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245330,"num_objects":2503,"num_object_clones":0,"num_object_copies":5006,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2503}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"28'2411","reported_seq":2452,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.359463+0000","last_change":"2024-04-22T08:27:06.406551+0000","last_active":"2024-04-22T08:27:33.359463+0000","last_peered":"2024-04-22T08:27:33.359463+0000","last_clean":"2024-04-22T08:26:38.731751+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:33.359463+0000","last_undegraded":"2024-04-22T08:27:06.388754+0000","last_fullsized":"2024-04-22T08:27:33.359463+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9852114,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":9624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2407}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"28'2439","reported_seq":2489,"reported_epoch":28,"state":"active+recovering","last_fresh":"2024-04-22T08:27:33.371304+0000","last_change":"2024-04-22T08:27:12.823560+0000","last_active":"2024-04-22T08:27:33.371304+0000","last_peered":"2024-04-22T08:27:33.371304+0000","last_clean":"2024-04-22T08:26:29.291065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:33.371304+0000","last_undegraded":"2024-04-22T08:27:33.371304+0000","last_fullsized":"2024-04-22T08:27:33.371304+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9982569,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":9749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"28'2581","reported_seq":2600,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.408164+0000","last_change":"2024-04-22T08:27:27.644057+0000","last_active":"2024-04-22T08:27:34.408164+0000","last_peered":"2024-04-22T08:27:34.408164+0000","last_clean":"2024-04-22T08:27:34.408164+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:34.408164+0000","last_undegraded":"2024-04-22T08:27:34.408164+0000","last_fullsized":"2024-04-22T08:27:34.408164+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10544338,"num_objects":2576,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"28'2422","reported_seq":2459,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.344361+0000","last_change":"2024-04-22T08:27:06.406525+0000","last_active":"2024-04-22T08:27:33.344361+0000","last_peered":"2024-04-22T08:27:33.344361+0000","last_clean":"2024-04-22T08:26:39.782656+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:33.344361+0000","last_undegraded":"2024-04-22T08:27:06.389578+0000","last_fullsized":"2024-04-22T08:27:33.344361+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2422,"log_dups_size":0,"ondisk_log_size":2422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9888978,"num_objects":2416,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":9660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2416}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"28'2452","reported_seq":2481,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.324110+0000","last_change":"2024-04-22T08:27:28.775685+0000","last_active":"2024-04-22T08:27:34.324110+0000","last_peered":"2024-04-22T08:27:34.324110+0000","last_clean":"2024-04-22T08:27:34.324110+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:34.324110+0000","last_undegraded":"2024-04-22T08:27:34.324110+0000","last_fullsized":"2024-04-22T08:27:34.324110+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024146,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":9791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"28'2528","reported_seq":2556,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:34.375833+0000","last_change":"2024-04-22T08:27:29.540465+0000","last_active":"2024-04-22T08:27:34.375833+0000","last_peered":"2024-04-22T08:27:34.375833+0000","last_clean":"2024-04-22T08:27:34.375833+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:34.375833+0000","last_undegraded":"2024-04-22T08:27:34.375833+0000","last_fullsized":"2024-04-22T08:27:34.375833+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2528,"log_dups_size":0,"ondisk_log_size":2528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10331346,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"28'2368","reported_seq":2408,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.370021+0000","last_change":"2024-04-22T08:27:06.406342+0000","last_active":"2024-04-22T08:27:33.370021+0000","last_peered":"2024-04-22T08:27:33.370021+0000","last_clean":"2024-04-22T08:26:32.531590+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:33.370021+0000","last_undegraded":"2024-04-22T08:27:06.389674+0000","last_fullsized":"2024-04-22T08:27:33.370021+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2368,"log_dups_size":0,"ondisk_log_size":2368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9691753,"num_objects":2367,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2367,"num_write_kb":9465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2367}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"28'2426","reported_seq":2465,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.361596+0000","last_change":"2024-04-22T08:27:06.407607+0000","last_active":"2024-04-22T08:27:33.361596+0000","last_peered":"2024-04-22T08:27:33.361596+0000","last_clean":"2024-04-22T08:26:39.622300+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:33.361596+0000","last_undegraded":"2024-04-22T08:27:06.390274+0000","last_fullsized":"2024-04-22T08:27:33.361596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2426,"log_dups_size":0,"ondisk_log_size":2426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9905979,"num_objects":2421,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2423,"num_write_kb":9676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2421}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"28'2395","reported_seq":2431,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.378805+0000","last_change":"2024-04-22T08:27:06.406945+0000","last_active":"2024-04-22T08:27:33.378805+0000","last_peered":"2024-04-22T08:27:33.378805+0000","last_clean":"2024-04-22T08:26:32.531788+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:33.378805+0000","last_undegraded":"2024-04-22T08:27:06.390162+0000","last_fullsized":"2024-04-22T08:27:33.378805+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9794770,"num_objects":2393,"num_object_clones":0,"num_object_copies":4786,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":9566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2393}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'2507","reported_seq":2546,"reported_epoch":28,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-22T08:27:33.385566+0000","last_change":"2024-04-22T08:27:06.404802+0000","last_active":"2024-04-22T08:27:33.385566+0000","last_peered":"2024-04-22T08:27:33.385566+0000","last_clean":"2024-04-22T08:26:39.211947+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:33.385566+0000","last_undegraded":"2024-04-22T08:27:06.384863+0000","last_fullsized":"2024-04-22T08:27:33.385566+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10221988,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":9987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2498}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161244028,"num_objects":39390,"num_object_clones":0,"num_object_copies":78780,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39417,"num_write_kb":157494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324100096,"data_stored":323981810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39444,"ondisk_log_size":39444,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270992,"kb_used_data":158980,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94100848,"statfs":{"total":96636764160,"available":96359268352,"internally_reserved":0,"allocated":162795520,"data_stored":162631396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114683599},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19500000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259068,"kb_used_data":151220,"kb_used_omap":4,"kb_used_meta":107835,"kb_avail":94112772,"statfs":{"total":96636764160,"available":96371478528,"internally_reserved":0,"allocated":154849280,"data_stored":154670744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":110423763},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.18099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159551488,"data_stored":159530614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164433920,"data_stored":164433920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:38.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 26 20 45930 45910 6.89666 4.49609 0.0121482 0.0113218 2024-04-22T08:27:39.813 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 27 20 46941 46921 6.78749 3.94922 0.0153369 0.011501 2024-04-22T08:27:40.814 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 28 20 48249 48229 6.72754 5.10938 0.00297404 0.0116023 2024-04-22T08:27:40.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:41.323 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-04-22T08:27:40.145627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188363644,"num_objects":46011,"num_object_clones":0,"num_object_copies":92022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46038,"num_write_kb":183978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46065,"ondisk_log_size":46065,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":582128,"kb_used_data":362272,"kb_used_omap":8,"kb_used_meta":219831,"kb_avail":188161552,"statfs":{"total":193273528320,"available":192677429248,"internally_reserved":0,"allocated":370966528,"data_stored":370620389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8798,"internal_metadata":225107362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,12,5,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":91418624,"num_objects":22319,"num_object_clones":0,"num_object_copies":44638,"num_objects_missing_on_primary":-25,"num_objects_missing":-25,"num_objects_degraded":-49,"num_objects_misplaced":0,"num_objects_unfound":-25,"num_objects_dirty":22319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22319,"num_write_kb":89276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004516"},"pg_stats":[{"pgid":"2.f","version":"28'2833","reported_seq":2873,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.301715+0000","last_change":"2024-04-22T08:27:06.404664+0000","last_active":"2024-04-22T08:27:38.301715+0000","last_peered":"2024-04-22T08:27:38.301715+0000","last_clean":"2024-04-22T08:27:38.301715+0000","last_became_active":"2024-04-22T08:27:06.403905+0000","last_became_peered":"2024-04-22T08:27:06.403905+0000","last_unstale":"2024-04-22T08:27:38.301715+0000","last_undegraded":"2024-04-22T08:27:38.301715+0000","last_fullsized":"2024-04-22T08:27:38.301715+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2833,"log_dups_size":0,"ondisk_log_size":2833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:18:25.202471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11603968,"num_objects":2833,"num_object_clones":0,"num_object_copies":5666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":11332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"28'2920","reported_seq":2973,"reported_epoch":28,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:27:39.402766+0000","last_change":"2024-04-22T08:27:38.878487+0000","last_active":"2024-04-22T08:27:39.402766+0000","last_peered":"2024-04-22T08:27:39.402766+0000","last_clean":"2024-04-22T08:27:39.402766+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:39.402766+0000","last_undegraded":"2024-04-22T08:27:39.402766+0000","last_fullsized":"2024-04-22T08:27:39.402766+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":949,"log_size":2920,"log_dups_size":0,"ondisk_log_size":2920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11936978,"num_objects":2916,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2918,"num_write_kb":11660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"28'2880","reported_seq":2900,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:39.402443+0000","last_change":"2024-04-22T08:27:24.544733+0000","last_active":"2024-04-22T08:27:39.402443+0000","last_peered":"2024-04-22T08:27:39.402443+0000","last_clean":"2024-04-22T08:27:39.402443+0000","last_became_active":"2024-04-22T08:27:06.398002+0000","last_became_peered":"2024-04-22T08:27:06.398002+0000","last_unstale":"2024-04-22T08:27:39.402443+0000","last_undegraded":"2024-04-22T08:27:39.402443+0000","last_fullsized":"2024-04-22T08:27:39.402443+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11780713,"num_objects":2877,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":11506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"28'2996","reported_seq":3016,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:39.380093+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:39.380093+0000","last_peered":"2024-04-22T08:27:39.380093+0000","last_clean":"2024-04-22T08:27:39.380093+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:39.380093+0000","last_undegraded":"2024-04-22T08:27:39.380093+0000","last_fullsized":"2024-04-22T08:27:39.380093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12271616,"num_objects":2996,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":11984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"28'2858","reported_seq":2914,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.335961+0000","last_change":"2024-04-22T08:27:37.656588+0000","last_active":"2024-04-22T08:27:38.335961+0000","last_peered":"2024-04-22T08:27:38.335961+0000","last_clean":"2024-04-22T08:27:38.335961+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:38.335961+0000","last_undegraded":"2024-04-22T08:27:38.335961+0000","last_fullsized":"2024-04-22T08:27:38.335961+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2858,"log_dups_size":0,"ondisk_log_size":2858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678930,"num_objects":2853,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"28'2912","reported_seq":2963,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.335970+0000","last_change":"2024-04-22T08:27:37.837462+0000","last_active":"2024-04-22T08:27:38.335970+0000","last_peered":"2024-04-22T08:27:38.335970+0000","last_clean":"2024-04-22T08:27:38.335970+0000","last_became_active":"2024-04-22T08:27:06.405845+0000","last_became_peered":"2024-04-22T08:27:06.405845+0000","last_unstale":"2024-04-22T08:27:38.335970+0000","last_undegraded":"2024-04-22T08:27:38.335970+0000","last_fullsized":"2024-04-22T08:27:38.335970+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2912,"log_dups_size":0,"ondisk_log_size":2912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908306,"num_objects":2909,"num_object_clones":0,"num_object_copies":5818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":11631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"28'2838","reported_seq":2889,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.328224+0000","last_change":"2024-04-22T08:27:37.968665+0000","last_active":"2024-04-22T08:27:38.328224+0000","last_peered":"2024-04-22T08:27:38.328224+0000","last_clean":"2024-04-22T08:27:38.328224+0000","last_became_active":"2024-04-22T08:27:06.405918+0000","last_became_peered":"2024-04-22T08:27:06.405918+0000","last_unstale":"2024-04-22T08:27:38.328224+0000","last_undegraded":"2024-04-22T08:27:38.328224+0000","last_fullsized":"2024-04-22T08:27:38.328224+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2838,"log_dups_size":0,"ondisk_log_size":2838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11601106,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":11332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"28'2814","reported_seq":2870,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.323748+0000","last_change":"2024-04-22T08:27:37.580884+0000","last_active":"2024-04-22T08:27:38.323748+0000","last_peered":"2024-04-22T08:27:38.323748+0000","last_clean":"2024-04-22T08:27:38.323748+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:38.323748+0000","last_undegraded":"2024-04-22T08:27:38.323748+0000","last_fullsized":"2024-04-22T08:27:38.323748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11518569,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404462+0000","last_change":"2024-04-22T08:27:06.404462+0000","last_active":"2024-04-22T08:27:06.404462+0000","last_peered":"2024-04-22T08:27:06.404462+0000","last_clean":"2024-04-22T08:27:06.404462+0000","last_became_active":"2024-04-22T08:27:06.403641+0000","last_became_peered":"2024-04-22T08:27:06.403641+0000","last_unstale":"2024-04-22T08:27:06.404462+0000","last_undegraded":"2024-04-22T08:27:06.404462+0000","last_fullsized":"2024-04-22T08:27:06.404462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:38:21.898988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"28'3007","reported_seq":3027,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:39.391953+0000","last_change":"2024-04-22T08:27:27.644057+0000","last_active":"2024-04-22T08:27:39.391953+0000","last_peered":"2024-04-22T08:27:39.391953+0000","last_clean":"2024-04-22T08:27:39.391953+0000","last_became_active":"2024-04-22T08:27:06.395973+0000","last_became_peered":"2024-04-22T08:27:06.395973+0000","last_unstale":"2024-04-22T08:27:39.391953+0000","last_undegraded":"2024-04-22T08:27:39.391953+0000","last_fullsized":"2024-04-22T08:27:39.391953+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12289234,"num_objects":3002,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3005,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.396348+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:06.396348+0000","last_peered":"2024-04-22T08:27:06.396348+0000","last_clean":"2024-04-22T08:27:06.396348+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:06.396348+0000","last_undegraded":"2024-04-22T08:27:06.396348+0000","last_fullsized":"2024-04-22T08:27:06.396348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"28'2834","reported_seq":2879,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.328259+0000","last_change":"2024-04-22T08:27:37.906404+0000","last_active":"2024-04-22T08:27:38.328259+0000","last_peered":"2024-04-22T08:27:38.328259+0000","last_clean":"2024-04-22T08:27:38.328259+0000","last_became_active":"2024-04-22T08:27:06.406051+0000","last_became_peered":"2024-04-22T08:27:06.406051+0000","last_unstale":"2024-04-22T08:27:38.328259+0000","last_undegraded":"2024-04-22T08:27:38.328259+0000","last_fullsized":"2024-04-22T08:27:38.328259+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11576530,"num_objects":2828,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":11308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.403006+0000","last_change":"2024-04-22T08:27:06.403006+0000","last_active":"2024-04-22T08:27:06.403006+0000","last_peered":"2024-04-22T08:27:06.403006+0000","last_clean":"2024-04-22T08:27:06.403006+0000","last_became_active":"2024-04-22T08:27:06.402395+0000","last_became_peered":"2024-04-22T08:27:06.402395+0000","last_unstale":"2024-04-22T08:27:06.403006+0000","last_undegraded":"2024-04-22T08:27:06.403006+0000","last_fullsized":"2024-04-22T08:27:06.403006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:37.029901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"28'2850","reported_seq":2879,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:39.394186+0000","last_change":"2024-04-22T08:27:28.775685+0000","last_active":"2024-04-22T08:27:39.394186+0000","last_peered":"2024-04-22T08:27:39.394186+0000","last_clean":"2024-04-22T08:27:39.394186+0000","last_became_active":"2024-04-22T08:27:06.388168+0000","last_became_peered":"2024-04-22T08:27:06.388168+0000","last_unstale":"2024-04-22T08:27:39.394186+0000","last_undegraded":"2024-04-22T08:27:39.394186+0000","last_fullsized":"2024-04-22T08:27:39.394186+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11654354,"num_objects":2847,"num_object_clones":0,"num_object_copies":5694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":11383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.395087+0000","last_change":"2024-04-22T08:27:06.395087+0000","last_active":"2024-04-22T08:27:06.395087+0000","last_peered":"2024-04-22T08:27:06.395087+0000","last_clean":"2024-04-22T08:27:06.395087+0000","last_became_active":"2024-04-22T08:27:06.393991+0000","last_became_peered":"2024-04-22T08:27:06.393991+0000","last_unstale":"2024-04-22T08:27:06.395087+0000","last_undegraded":"2024-04-22T08:27:06.395087+0000","last_fullsized":"2024-04-22T08:27:06.395087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:59:14.245651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"28'2936","reported_seq":2964,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:39.380081+0000","last_change":"2024-04-22T08:27:29.540465+0000","last_active":"2024-04-22T08:27:39.380081+0000","last_peered":"2024-04-22T08:27:39.380081+0000","last_clean":"2024-04-22T08:27:39.380081+0000","last_became_active":"2024-04-22T08:27:06.392414+0000","last_became_peered":"2024-04-22T08:27:06.392414+0000","last_unstale":"2024-04-22T08:27:39.380081+0000","last_undegraded":"2024-04-22T08:27:39.380081+0000","last_fullsized":"2024-04-22T08:27:39.380081+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12002514,"num_objects":2932,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":11723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"28'2771","reported_seq":2819,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.329804+0000","last_change":"2024-04-22T08:27:37.758654+0000","last_active":"2024-04-22T08:27:38.329804+0000","last_peered":"2024-04-22T08:27:38.329804+0000","last_clean":"2024-04-22T08:27:38.329804+0000","last_became_active":"2024-04-22T08:27:06.405751+0000","last_became_peered":"2024-04-22T08:27:06.405751+0000","last_unstale":"2024-04-22T08:27:38.329804+0000","last_undegraded":"2024-04-22T08:27:38.329804+0000","last_fullsized":"2024-04-22T08:27:38.329804+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11342441,"num_objects":2770,"num_object_clones":0,"num_object_copies":5540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":11077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404188+0000","last_change":"2024-04-22T08:27:06.404188+0000","last_active":"2024-04-22T08:27:06.404188+0000","last_peered":"2024-04-22T08:27:06.404188+0000","last_clean":"2024-04-22T08:27:06.404188+0000","last_became_active":"2024-04-22T08:27:06.403542+0000","last_became_peered":"2024-04-22T08:27:06.403542+0000","last_unstale":"2024-04-22T08:27:06.404188+0000","last_undegraded":"2024-04-22T08:27:06.404188+0000","last_fullsized":"2024-04-22T08:27:06.404188+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:31:17.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"28'2875","reported_seq":2928,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.327998+0000","last_change":"2024-04-22T08:27:38.103713+0000","last_active":"2024-04-22T08:27:38.327998+0000","last_peered":"2024-04-22T08:27:38.327998+0000","last_clean":"2024-04-22T08:27:38.327998+0000","last_became_active":"2024-04-22T08:27:06.407085+0000","last_became_peered":"2024-04-22T08:27:06.407085+0000","last_unstale":"2024-04-22T08:27:38.327998+0000","last_undegraded":"2024-04-22T08:27:38.327998+0000","last_fullsized":"2024-04-22T08:27:38.327998+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11745083,"num_objects":2870,"num_object_clones":0,"num_object_copies":5740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2872,"num_write_kb":11472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398585+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:06.398585+0000","last_peered":"2024-04-22T08:27:06.398585+0000","last_clean":"2024-04-22T08:27:06.398585+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:06.398585+0000","last_undegraded":"2024-04-22T08:27:06.398585+0000","last_fullsized":"2024-04-22T08:27:06.398585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"28'2800","reported_seq":2850,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.286750+0000","last_change":"2024-04-22T08:27:38.008702+0000","last_active":"2024-04-22T08:27:38.286750+0000","last_peered":"2024-04-22T08:27:38.286750+0000","last_clean":"2024-04-22T08:27:38.286750+0000","last_became_active":"2024-04-22T08:27:06.406288+0000","last_became_peered":"2024-04-22T08:27:06.406288+0000","last_unstale":"2024-04-22T08:27:38.286750+0000","last_undegraded":"2024-04-22T08:27:38.286750+0000","last_fullsized":"2024-04-22T08:27:38.286750+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2800,"log_dups_size":0,"ondisk_log_size":2800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11453650,"num_objects":2798,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":11186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.404609+0000","last_change":"2024-04-22T08:27:06.404609+0000","last_active":"2024-04-22T08:27:06.404609+0000","last_peered":"2024-04-22T08:27:06.404609+0000","last_clean":"2024-04-22T08:27:06.404609+0000","last_became_active":"2024-04-22T08:27:06.404053+0000","last_became_peered":"2024-04-22T08:27:06.404053+0000","last_unstale":"2024-04-22T08:27:06.404609+0000","last_undegraded":"2024-04-22T08:27:06.404609+0000","last_fullsized":"2024-04-22T08:27:06.404609+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:46:33.184549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:06.398517+0000","last_change":"2024-04-22T08:27:06.398517+0000","last_active":"2024-04-22T08:27:06.398517+0000","last_peered":"2024-04-22T08:27:06.398517+0000","last_clean":"2024-04-22T08:27:06.398517+0000","last_became_active":"2024-04-22T08:27:06.396445+0000","last_became_peered":"2024-04-22T08:27:06.396445+0000","last_unstale":"2024-04-22T08:27:06.398517+0000","last_undegraded":"2024-04-22T08:27:06.398517+0000","last_fullsized":"2024-04-22T08:27:06.398517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:19:12.018171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'2941","reported_seq":2996,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T08:27:38.343060+0000","last_change":"2024-04-22T08:27:37.734534+0000","last_active":"2024-04-22T08:27:38.343060+0000","last_peered":"2024-04-22T08:27:38.343060+0000","last_clean":"2024-04-22T08:27:38.343060+0000","last_became_active":"2024-04-22T08:27:06.404039+0000","last_became_peered":"2024-04-22T08:27:06.404039+0000","last_unstale":"2024-04-22T08:27:38.343060+0000","last_undegraded":"2024-04-22T08:27:38.343060+0000","last_fullsized":"2024-04-22T08:27:38.343060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":2941,"log_dups_size":0,"ondisk_log_size":2941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11999652,"num_objects":2933,"num_object_clones":0,"num_object_copies":5866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2938,"num_write_kb":11723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":188363644,"num_objects":46011,"num_object_clones":0,"num_object_copies":92022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46038,"num_write_kb":183978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377954304,"data_stored":377849934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46065,"ondisk_log_size":46065,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117003,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296304,"kb_used_data":184296,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94075536,"statfs":{"total":96636764160,"available":96333348864,"internally_reserved":0,"allocated":188719104,"data_stored":188551501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114683599},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5599999999999996}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313231,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285824,"kb_used_data":177976,"kb_used_omap":4,"kb_used_meta":107835,"kb_avail":94086016,"statfs":{"total":96636764160,"available":96344080384,"internally_reserved":0,"allocated":182247424,"data_stored":182068888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":110423763},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,0,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186621952,"data_stored":186618473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191217664,"data_stored":191214185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:41.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-04-22T08:27:41.363 INFO:tasks.util.rados:rados -p unfounddel_pool get new_1 - 2024-04-22T08:27:41.363 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_1 - 2024-04-22T08:27:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_1: (2) No such file or directory 2024-04-22T08:27:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.476 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_1 - 2024-04-22T08:27:41.476 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_1 - 2024-04-22T08:27:41.556 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_1: (2) No such file or directory 2024-04-22T08:27:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.575 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_1 - 2024-04-22T08:27:41.576 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_1 - 2024-04-22T08:27:41.659 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_1: (2) No such file or directory 2024-04-22T08:27:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.664 INFO:tasks.util.rados:rados -p unfounddel_pool get new_2 - 2024-04-22T08:27:41.664 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_2 - 2024-04-22T08:27:41.727 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_2: (2) No such file or directory 2024-04-22T08:27:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.734 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_2 - 2024-04-22T08:27:41.734 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_2 - 2024-04-22T08:27:41.811 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_2: (2) No such file or directory 2024-04-22T08:27:41.814 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 29 20 49364 49344 6.64555 4.35547 0.0290642 0.0117475 2024-04-22T08:27:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.817 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_2 - 2024-04-22T08:27:41.818 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_2 - 2024-04-22T08:27:41.918 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_2: (2) No such file or directory 2024-04-22T08:27:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:41.924 INFO:tasks.util.rados:rados -p unfounddel_pool get new_3 - 2024-04-22T08:27:41.924 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_3 - 2024-04-22T08:27:42.025 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_3: (2) No such file or directory 2024-04-22T08:27:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.033 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_3 - 2024-04-22T08:27:42.033 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_3 - 2024-04-22T08:27:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_3: (2) No such file or directory 2024-04-22T08:27:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.121 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_3 - 2024-04-22T08:27:42.121 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_3 - 2024-04-22T08:27:42.228 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_3: (2) No such file or directory 2024-04-22T08:27:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.233 INFO:tasks.util.rados:rados -p unfounddel_pool get new_4 - 2024-04-22T08:27:42.234 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_4 - 2024-04-22T08:27:42.319 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_4: (2) No such file or directory 2024-04-22T08:27:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.420 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_4 - 2024-04-22T08:27:42.420 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_4 - 2024-04-22T08:27:42.502 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_4: (2) No such file or directory 2024-04-22T08:27:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.514 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_4 - 2024-04-22T08:27:42.514 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_4 - 2024-04-22T08:27:42.598 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_4: (2) No such file or directory 2024-04-22T08:27:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.611 INFO:tasks.util.rados:rados -p unfounddel_pool get new_5 - 2024-04-22T08:27:42.611 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_5 - 2024-04-22T08:27:42.671 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_5: (2) No such file or directory 2024-04-22T08:27:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.689 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_5 - 2024-04-22T08:27:42.689 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_5 - 2024-04-22T08:27:42.782 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_5: (2) No such file or directory 2024-04-22T08:27:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.790 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_5 - 2024-04-22T08:27:42.790 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_5 - 2024-04-22T08:27:42.815 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 30 20 50623 50603 6.58778 4.91797 0.0271787 0.0118524 2024-04-22T08:27:42.889 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_5: (2) No such file or directory 2024-04-22T08:27:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:42.897 INFO:tasks.util.rados:rados -p unfounddel_pool get new_6 - 2024-04-22T08:27:42.898 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_6 - 2024-04-22T08:27:43.037 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_6: (2) No such file or directory 2024-04-22T08:27:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.043 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_6 - 2024-04-22T08:27:43.044 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_6 - 2024-04-22T08:27:43.148 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_6: (2) No such file or directory 2024-04-22T08:27:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.154 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_6 - 2024-04-22T08:27:43.154 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_6 - 2024-04-22T08:27:43.232 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_6: (2) No such file or directory 2024-04-22T08:27:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.246 INFO:tasks.util.rados:rados -p unfounddel_pool get new_7 - 2024-04-22T08:27:43.246 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_7 - 2024-04-22T08:27:43.352 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_7: (2) No such file or directory 2024-04-22T08:27:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.364 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_7 - 2024-04-22T08:27:43.364 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_7 - 2024-04-22T08:27:43.458 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_7: (2) No such file or directory 2024-04-22T08:27:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.471 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_7 - 2024-04-22T08:27:43.471 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_7 - 2024-04-22T08:27:43.568 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_7: (2) No such file or directory 2024-04-22T08:27:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.578 INFO:tasks.util.rados:rados -p unfounddel_pool get new_8 - 2024-04-22T08:27:43.578 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_8 - 2024-04-22T08:27:43.646 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_8: (2) No such file or directory 2024-04-22T08:27:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.656 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_8 - 2024-04-22T08:27:43.656 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_8 - 2024-04-22T08:27:43.720 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_8: (2) No such file or directory 2024-04-22T08:27:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.732 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_8 - 2024-04-22T08:27:43.732 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_8 - 2024-04-22T08:27:43.816 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 31 20 51812 51792 6.52494 4.64453 0.0152906 0.0119656 2024-04-22T08:27:43.829 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_8: (2) No such file or directory 2024-04-22T08:27:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.839 INFO:tasks.util.rados:rados -p unfounddel_pool get new_9 - 2024-04-22T08:27:43.840 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_9 - 2024-04-22T08:27:43.945 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/new_9: (2) No such file or directory 2024-04-22T08:27:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:43.950 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_9 - 2024-04-22T08:27:43.950 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_9 - 2024-04-22T08:27:44.013 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existed_9: (2) No such file or directory 2024-04-22T08:27:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:44.017 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_9 - 2024-04-22T08:27:44.018 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_9 - 2024-04-22T08:27:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:error getting unfounddel_pool/existing_9: (2) No such file or directory 2024-04-22T08:27:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:27:44.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-04-22T08:27:44.816 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 32 20 52960 52940 6.46117 4.48438 0.0139792 0.0120833 2024-04-22T08:27:44.897 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-04-22T08:27:44.898+0000 7f403222d640 -1 mon.c@2(leader).osd e28 definitely_dead 0 2024-04-22T08:27:45.803 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.1. 2024-04-22T08:27:45.816 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 33 20 53959 53939 6.38363 3.90234 0.0142235 0.0122169 2024-04-22T08:27:45.835 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T08:27:45.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-22T08:27:45.837 INFO:tasks.ceph.osd.1:Started 2024-04-22T08:27:45.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:no valid command found; 10 closest matches: 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:0 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:1 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:2 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:abort 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:assert 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:config diff 2024-04-22T08:27:45.937 INFO:teuthology.orchestra.run.smithi049.stderr:config diff get 2024-04-22T08:27:45.938 INFO:teuthology.orchestra.run.smithi049.stderr:config get 2024-04-22T08:27:45.938 INFO:teuthology.orchestra.run.smithi049.stderr:config help [] 2024-04-22T08:27:45.938 INFO:teuthology.orchestra.run.smithi049.stderr:config set ... 2024-04-22T08:27:45.938 INFO:teuthology.orchestra.run.smithi049.stderr:admin_socket: invalid command 2024-04-22T08:27:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T08:27:45.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T08:27:46.430 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:27:46.430+0000 7fa6f517d7c0 -1 Falling back to public interface 2024-04-22T08:27:46.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 34 20 53966 53946 6.1967 0.0273438 0.00573438 0.0122194 2024-04-22T08:27:47.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 35 20 53966 53946 6.01966 0 - 0.0122194 2024-04-22T08:27:48.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 36 20 55111 55091 5.97668 2.23633 0.0122881 0.0130663 2024-04-22T08:27:49.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 37 20 56362 56342 5.94721 4.88672 0.0119709 0.0131276 2024-04-22T08:27:50.653 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:27:50.654+0000 7fa6f517d7c0 -1 osd.1 22 log_to_monitors true 2024-04-22T08:27:50.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 38 20 57499 57479 5.90758 4.44141 0.020422 0.0132178 2024-04-22T08:27:50.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:no valid command found; 10 closest matches: 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:0 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:1 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:2 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:abort 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:assert 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:bluefs debug_inject_read_zeros 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:bluefs files list 2024-04-22T08:27:51.291 INFO:teuthology.orchestra.run.smithi049.stderr:bluefs stats 2024-04-22T08:27:51.292 INFO:teuthology.orchestra.run.smithi049.stderr:bluestore allocator dump block 2024-04-22T08:27:51.292 INFO:teuthology.orchestra.run.smithi049.stderr:bluestore allocator fragmentation block 2024-04-22T08:27:51.292 INFO:teuthology.orchestra.run.smithi049.stderr:admin_socket: invalid command 2024-04-22T08:27:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T08:27:51.296 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T08:27:51.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 39 20 58545 58525 5.86087 4.08594 0.0231479 0.0133239 2024-04-22T08:27:52.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:27:52.820988+0000 min lat: 0.00109492 max lat: 2.36078 avg lat: 0.0134451 2024-04-22T08:27:52.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:27:52.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 40 20 59482 59462 5.80585 3.66016 0.00303183 0.0134451 2024-04-22T08:27:53.060 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-04-22T08:27:53.062+0000 7fa6e8f44640 -1 osd.1 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:27:53.817 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 41 20 60535 60515 5.76456 4.11328 0.0115157 0.0135448 2024-04-22T08:27:54.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 42 20 60840 60820 5.65568 1.19141 0.00313658 0.013566 2024-04-22T08:27:55.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 43 20 60840 60820 5.52415 0 - 0.013566 2024-04-22T08:27:56.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T08:27:56.528 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:27:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-22T08:27:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-22T08:27:56.528 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:27:56.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.1 to be up 2024-04-22T08:27:56.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:27:56.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 44 20 60895 60875 5.40349 0.107422 0.0080621 0.0143983 2024-04-22T08:27:57.189 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:57.189 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":35,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:27:56.076664+0000","last_up_change":"2024-04-22T08:27:54.066710+0000","last_in_change":"2024-04-22T08:27:45.797437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-04-22T08:26:26.331332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":21,"up_thru":34,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":29827}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":29827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":29827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":29827}]},"public_addr":"172.21.15.49:6800/29827","cluster_addr":"172.21.15.49:6801/29827","heartbeat_back_addr":"172.21.15.49:6803/29827","heartbeat_front_addr":"172.21.15.49:6802/29827","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":22,"up_from":33,"up_thru":34,"down_at":23,"lost_at":23,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":34075}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":34075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":34075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":34075}]},"public_addr":"172.21.15.49:6808/34075","cluster_addr":"172.21.15.49:6809/34075","heartbeat_back_addr":"172.21.15.49:6811/34075","heartbeat_front_addr":"172.21.15.49:6810/34075","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":14,"up_from":27,"up_thru":34,"down_at":15,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":31087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":31087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":31087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":31087}]},"public_addr":"172.21.15.49:6804/31087","cluster_addr":"172.21.15.49:6805/31087","heartbeat_back_addr":"172.21.15.49:6807/31087","heartbeat_front_addr":"172.21.15.49:6806/31087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":18},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":15}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.0","osds":[2]},{"pgid":"2.1","osds":[2]},{"pgid":"2.2","osds":[0]},{"pgid":"2.3","osds":[2]},{"pgid":"2.4","osds":[0]},{"pgid":"2.5","osds":[0]},{"pgid":"2.6","osds":[0]},{"pgid":"2.7","osds":[0]},{"pgid":"2.9","osds":[0]},{"pgid":"2.a","osds":[0]},{"pgid":"2.d","osds":[2]},{"pgid":"2.f","osds":[0]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:27:57.246 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.1 is up 2024-04-22T08:27:57.246 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-04-22T08:27:57.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:57.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 45 20 61522 61502 5.33784 2.44922 0.0664595 0.014626 2024-04-22T08:27:57.965 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:57.969 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:58.017 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-22T08:27:56.149505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":36,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":800004,"kb_used_data":480424,"kb_used_omap":12,"kb_used_meta":319539,"kb_avail":282315516,"statfs":{"total":289910292480,"available":289091088384,"internally_reserved":0,"allocated":491954176,"data_stored":491357267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":327208046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,7,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21513820,"num_objects":5249,"num_object_clones":0,"num_object_copies":10498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11253,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1921,"num_write_kb":-7663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.940339"},"pg_stats":[{"pgid":"2.f","version":"32'3754","reported_seq":3807,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.390426+0000","last_change":"2024-04-22T08:27:47.905466+0000","last_active":"2024-04-22T08:27:53.390426+0000","last_peered":"2024-04-22T08:27:53.390426+0000","last_clean":"2024-04-22T08:27:53.390426+0000","last_became_active":"2024-04-22T08:27:47.898939+0000","last_became_peered":"2024-04-22T08:27:47.898939+0000","last_unstale":"2024-04-22T08:27:53.390426+0000","last_undegraded":"2024-04-22T08:27:53.390426+0000","last_fullsized":"2024-04-22T08:27:53.390426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:27.633840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15376384,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":15016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3754}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"33'3819","reported_seq":3929,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.094077+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:54.094077+0000","last_peered":"2024-04-22T08:27:54.094077+0000","last_clean":"2024-04-22T08:27:54.094077+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:54.094077+0000","last_undegraded":"2024-04-22T08:27:54.094077+0000","last_fullsized":"2024-04-22T08:27:54.094077+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:19.957180+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":3500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"32'3759","reported_seq":3870,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084517+0000","last_change":"2024-04-22T08:27:54.084606+0000","last_active":"2024-04-22T08:27:54.084517+0000","last_peered":"2024-04-22T08:27:54.071080+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:47.865324+0000","last_became_peered":"2024-04-22T08:27:47.865324+0000","last_unstale":"2024-04-22T08:27:54.084517+0000","last_undegraded":"2024-04-22T08:27:54.084517+0000","last_fullsized":"2024-04-22T08:27:54.084517+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:08.298486+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15380480,"num_objects":3755,"num_object_clones":0,"num_object_copies":7510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":586,"num_write_kb":2344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"32'3888","reported_seq":3913,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.075301+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:54.075301+0000","last_peered":"2024-04-22T08:27:54.075301+0000","last_clean":"2024-04-22T08:27:54.075301+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:54.075301+0000","last_undegraded":"2024-04-22T08:27:54.075301+0000","last_fullsized":"2024-04-22T08:27:54.075301+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":3888,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"32'3770","reported_seq":3902,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.372138+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:27:53.372138+0000","last_peered":"2024-04-22T08:27:53.372138+0000","last_clean":"2024-04-22T08:27:53.372138+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:53.372138+0000","last_undegraded":"2024-04-22T08:27:53.372138+0000","last_fullsized":"2024-04-22T08:27:53.372138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:13.316934+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15413248,"num_objects":3763,"num_object_clones":0,"num_object_copies":7526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":3448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"32'3841","reported_seq":3989,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.380648+0000","last_change":"2024-04-22T08:27:47.962032+0000","last_active":"2024-04-22T08:27:53.380648+0000","last_peered":"2024-04-22T08:27:53.380648+0000","last_clean":"2024-04-22T08:27:53.380648+0000","last_became_active":"2024-04-22T08:27:47.912071+0000","last_became_peered":"2024-04-22T08:27:47.912071+0000","last_unstale":"2024-04-22T08:27:53.380648+0000","last_undegraded":"2024-04-22T08:27:53.380648+0000","last_fullsized":"2024-04-22T08:27:53.380648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:24.363832+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15712256,"num_objects":3836,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3836,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"32'3738","reported_seq":3887,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.365338+0000","last_change":"2024-04-22T08:27:52.931316+0000","last_active":"2024-04-22T08:27:53.365338+0000","last_peered":"2024-04-22T08:27:53.365338+0000","last_clean":"2024-04-22T08:27:53.365338+0000","last_became_active":"2024-04-22T08:27:47.865037+0000","last_became_peered":"2024-04-22T08:27:47.865037+0000","last_unstale":"2024-04-22T08:27:53.365338+0000","last_undegraded":"2024-04-22T08:27:53.365338+0000","last_fullsized":"2024-04-22T08:27:53.365338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:21:05.715899+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15286272,"num_objects":3732,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3732,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"32'3707","reported_seq":3845,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.371591+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:27:53.371591+0000","last_peered":"2024-04-22T08:27:53.371591+0000","last_clean":"2024-04-22T08:27:53.371591+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:53.371591+0000","last_undegraded":"2024-04-22T08:27:53.371591+0000","last_fullsized":"2024-04-22T08:27:53.371591+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:42:06.940931+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15175680,"num_objects":3705,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":1420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.893664+0000","last_change":"2024-04-22T08:27:47.884472+0000","last_active":"2024-04-22T08:27:47.893664+0000","last_peered":"2024-04-22T08:27:47.893664+0000","last_clean":"2024-04-22T08:27:47.893664+0000","last_became_active":"2024-04-22T08:27:47.880654+0000","last_became_peered":"2024-04-22T08:27:47.880654+0000","last_unstale":"2024-04-22T08:27:47.893664+0000","last_undegraded":"2024-04-22T08:27:47.893664+0000","last_fullsized":"2024-04-22T08:27:47.893664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:52.507060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"32'3953","reported_seq":4064,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.082918+0000","last_change":"2024-04-22T08:27:54.083074+0000","last_active":"2024-04-22T08:27:54.082918+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.895678+0000","last_became_peered":"2024-04-22T08:27:47.895678+0000","last_unstale":"2024-04-22T08:27:54.082918+0000","last_undegraded":"2024-04-22T08:27:54.082918+0000","last_fullsized":"2024-04-22T08:27:54.082918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":3953,"log_dups_size":0,"ondisk_log_size":3953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15926482,"num_objects":3890,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":15556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.212030+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:51.212030+0000","last_peered":"2024-04-22T08:27:51.212030+0000","last_clean":"2024-04-22T08:27:51.212030+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:51.212030+0000","last_undegraded":"2024-04-22T08:27:51.212030+0000","last_fullsized":"2024-04-22T08:27:51.212030+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"32'3731","reported_seq":3795,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.330904+0000","last_change":"2024-04-22T08:27:47.975278+0000","last_active":"2024-04-22T08:27:53.330904+0000","last_peered":"2024-04-22T08:27:53.330904+0000","last_clean":"2024-04-22T08:27:53.330904+0000","last_became_active":"2024-04-22T08:27:47.907574+0000","last_became_peered":"2024-04-22T08:27:47.907574+0000","last_unstale":"2024-04-22T08:27:53.330904+0000","last_undegraded":"2024-04-22T08:27:53.330904+0000","last_fullsized":"2024-04-22T08:27:53.330904+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15250642,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":14896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.921458+0000","last_change":"2024-04-22T08:27:47.907718+0000","last_active":"2024-04-22T08:27:47.921458+0000","last_peered":"2024-04-22T08:27:47.921458+0000","last_clean":"2024-04-22T08:27:47.921458+0000","last_became_active":"2024-04-22T08:27:47.904280+0000","last_became_peered":"2024-04-22T08:27:47.904280+0000","last_unstale":"2024-04-22T08:27:47.921458+0000","last_undegraded":"2024-04-22T08:27:47.921458+0000","last_fullsized":"2024-04-22T08:27:47.921458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:54.701862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"32'3755","reported_seq":3889,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084354+0000","last_change":"2024-04-22T08:27:54.084505+0000","last_active":"2024-04-22T08:27:54.084354+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.891656+0000","last_became_peered":"2024-04-22T08:27:47.891656+0000","last_unstale":"2024-04-22T08:27:54.084354+0000","last_undegraded":"2024-04-22T08:27:54.084354+0000","last_fullsized":"2024-04-22T08:27:54.084354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:14:17.011512+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15360000,"num_objects":3750,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":28,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.082499+0000","last_change":"2024-04-22T08:27:54.082640+0000","last_active":"2024-04-22T08:27:54.082499+0000","last_peered":"2024-04-22T08:27:51.208056+0000","last_clean":"2024-04-22T08:27:51.208056+0000","last_became_active":"2024-04-22T08:27:47.865568+0000","last_became_peered":"2024-04-22T08:27:47.865568+0000","last_unstale":"2024-04-22T08:27:54.082499+0000","last_undegraded":"2024-04-22T08:27:54.082499+0000","last_fullsized":"2024-04-22T08:27:54.082499+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:28:16.339720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"32'3809","reported_seq":3852,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.083001+0000","last_change":"2024-04-22T08:27:54.083165+0000","last_active":"2024-04-22T08:27:54.083001+0000","last_peered":"2024-04-22T08:27:54.074517+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:47.897092+0000","last_became_peered":"2024-04-22T08:27:47.897092+0000","last_unstale":"2024-04-22T08:27:54.083001+0000","last_undegraded":"2024-04-22T08:27:54.083001+0000","last_fullsized":"2024-04-22T08:27:54.083001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15578322,"num_objects":3805,"num_object_clones":0,"num_object_copies":7610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":15215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"32'3673","reported_seq":3738,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.412759+0000","last_change":"2024-04-22T08:27:47.943170+0000","last_active":"2024-04-22T08:27:53.412759+0000","last_peered":"2024-04-22T08:27:53.412759+0000","last_clean":"2024-04-22T08:27:53.412759+0000","last_became_active":"2024-04-22T08:27:47.916592+0000","last_became_peered":"2024-04-22T08:27:47.916592+0000","last_unstale":"2024-04-22T08:27:53.412759+0000","last_undegraded":"2024-04-22T08:27:53.412759+0000","last_fullsized":"2024-04-22T08:27:53.412759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15037033,"num_objects":3672,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3672,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":14685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3672}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.909071+0000","last_change":"2024-04-22T08:27:47.908352+0000","last_active":"2024-04-22T08:27:47.909071+0000","last_peered":"2024-04-22T08:27:47.909071+0000","last_clean":"2024-04-22T08:27:47.909071+0000","last_became_active":"2024-04-22T08:27:47.907923+0000","last_became_peered":"2024-04-22T08:27:47.907923+0000","last_unstale":"2024-04-22T08:27:47.909071+0000","last_undegraded":"2024-04-22T08:27:47.909071+0000","last_fullsized":"2024-04-22T08:27:47.909071+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:33:20.028960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"32'3798","reported_seq":3930,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.405662+0000","last_change":"2024-04-22T08:27:51.614530+0000","last_active":"2024-04-22T08:27:53.405662+0000","last_peered":"2024-04-22T08:27:53.405662+0000","last_clean":"2024-04-22T08:27:53.405662+0000","last_became_active":"2024-04-22T08:27:47.910831+0000","last_became_peered":"2024-04-22T08:27:47.910831+0000","last_unstale":"2024-04-22T08:27:53.405662+0000","last_undegraded":"2024-04-22T08:27:53.405662+0000","last_fullsized":"2024-04-22T08:27:53.405662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2849,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15525691,"num_objects":3793,"num_object_clones":0,"num_object_copies":7586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3793,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":15164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.208360+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:51.208360+0000","last_peered":"2024-04-22T08:27:51.208360+0000","last_clean":"2024-04-22T08:27:51.208360+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:51.208360+0000","last_undegraded":"2024-04-22T08:27:51.208360+0000","last_fullsized":"2024-04-22T08:27:51.208360+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"32'3661","reported_seq":3723,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.395008+0000","last_change":"2024-04-22T08:27:47.884233+0000","last_active":"2024-04-22T08:27:53.395008+0000","last_peered":"2024-04-22T08:27:53.395008+0000","last_clean":"2024-04-22T08:27:53.395008+0000","last_became_active":"2024-04-22T08:27:47.864504+0000","last_became_peered":"2024-04-22T08:27:47.864504+0000","last_unstale":"2024-04-22T08:27:53.395008+0000","last_undegraded":"2024-04-22T08:27:53.395008+0000","last_fullsized":"2024-04-22T08:27:53.395008+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980306,"num_objects":3659,"num_object_clones":0,"num_object_copies":7318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":14630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.895222+0000","last_change":"2024-04-22T08:27:47.894716+0000","last_active":"2024-04-22T08:27:47.895222+0000","last_peered":"2024-04-22T08:27:47.895222+0000","last_clean":"2024-04-22T08:27:47.895222+0000","last_became_active":"2024-04-22T08:27:47.894245+0000","last_became_peered":"2024-04-22T08:27:47.894245+0000","last_unstale":"2024-04-22T08:27:47.895222+0000","last_undegraded":"2024-04-22T08:27:47.895222+0000","last_fullsized":"2024-04-22T08:27:47.895222+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:01.575183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":28,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.083113+0000","last_change":"2024-04-22T08:27:54.083360+0000","last_active":"2024-04-22T08:27:54.083113+0000","last_peered":"2024-04-22T08:27:51.208244+0000","last_clean":"2024-04-22T08:27:51.208244+0000","last_became_active":"2024-04-22T08:27:47.911130+0000","last_became_peered":"2024-04-22T08:27:47.911130+0000","last_unstale":"2024-04-22T08:27:54.083113+0000","last_undegraded":"2024-04-22T08:27:54.083113+0000","last_fullsized":"2024-04-22T08:27:54.083113+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:40:35.524437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"32'3838","reported_seq":3944,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.406331+0000","last_change":"2024-04-22T08:27:52.631343+0000","last_active":"2024-04-22T08:27:53.406331+0000","last_peered":"2024-04-22T08:27:53.406331+0000","last_clean":"2024-04-22T08:27:53.406331+0000","last_became_active":"2024-04-22T08:27:47.924447+0000","last_became_peered":"2024-04-22T08:27:47.924447+0000","last_unstale":"2024-04-22T08:27:53.406331+0000","last_undegraded":"2024-04-22T08:27:53.406331+0000","last_fullsized":"2024-04-22T08:27:53.406331+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":1599,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15673764,"num_objects":3830,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3830,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495357952,"data_stored":495253582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390700,"kb_used_data":242596,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93981140,"statfs":{"total":96636764160,"available":96236687360,"internally_reserved":0,"allocated":248418304,"data_stored":248231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":296,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":303104,"data_stored":72469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":28176076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":21,"seq":90194313234,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381480,"kb_used_data":237532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93990360,"statfs":{"total":96636764160,"available":96246128640,"internally_reserved":0,"allocated":243232768,"data_stored":243052862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246091776,"data_stored":246088297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249151488,"data_stored":249148009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:58.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:58.803 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:58.818 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 46 20 62155 62135 5.27556 2.47266 0.0237667 0.0147997 2024-04-22T08:27:58.888 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-04-22T08:27:58.149986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":39,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":804236,"kb_used_data":480548,"kb_used_omap":12,"kb_used_meta":323635,"kb_avail":282311284,"statfs":{"total":289910292480,"available":289086754816,"internally_reserved":0,"allocated":492081152,"data_stored":491437325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":331402350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,7,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21513820,"num_objects":5249,"num_object_clones":0,"num_object_copies":10498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11253,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1921,"num_write_kb":-7663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001302"},"pg_stats":[{"pgid":"2.f","version":"32'3754","reported_seq":3807,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.390426+0000","last_change":"2024-04-22T08:27:47.905466+0000","last_active":"2024-04-22T08:27:53.390426+0000","last_peered":"2024-04-22T08:27:53.390426+0000","last_clean":"2024-04-22T08:27:53.390426+0000","last_became_active":"2024-04-22T08:27:47.898939+0000","last_became_peered":"2024-04-22T08:27:47.898939+0000","last_unstale":"2024-04-22T08:27:53.390426+0000","last_undegraded":"2024-04-22T08:27:53.390426+0000","last_fullsized":"2024-04-22T08:27:53.390426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:27.633840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15376384,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":15016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3754}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"33'3819","reported_seq":3929,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.094077+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:54.094077+0000","last_peered":"2024-04-22T08:27:54.094077+0000","last_clean":"2024-04-22T08:27:54.094077+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:54.094077+0000","last_undegraded":"2024-04-22T08:27:54.094077+0000","last_fullsized":"2024-04-22T08:27:54.094077+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:19.957180+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":3500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"32'3759","reported_seq":3870,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084517+0000","last_change":"2024-04-22T08:27:54.084606+0000","last_active":"2024-04-22T08:27:54.084517+0000","last_peered":"2024-04-22T08:27:54.071080+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:47.865324+0000","last_became_peered":"2024-04-22T08:27:47.865324+0000","last_unstale":"2024-04-22T08:27:54.084517+0000","last_undegraded":"2024-04-22T08:27:54.084517+0000","last_fullsized":"2024-04-22T08:27:54.084517+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:08.298486+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15380480,"num_objects":3755,"num_object_clones":0,"num_object_copies":7510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":586,"num_write_kb":2344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"32'3888","reported_seq":3913,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.075301+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:54.075301+0000","last_peered":"2024-04-22T08:27:54.075301+0000","last_clean":"2024-04-22T08:27:54.075301+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:54.075301+0000","last_undegraded":"2024-04-22T08:27:54.075301+0000","last_fullsized":"2024-04-22T08:27:54.075301+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":3888,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"32'3770","reported_seq":3902,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.372138+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:27:53.372138+0000","last_peered":"2024-04-22T08:27:53.372138+0000","last_clean":"2024-04-22T08:27:53.372138+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:53.372138+0000","last_undegraded":"2024-04-22T08:27:53.372138+0000","last_fullsized":"2024-04-22T08:27:53.372138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:13.316934+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15413248,"num_objects":3763,"num_object_clones":0,"num_object_copies":7526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":3448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"32'3841","reported_seq":3989,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.380648+0000","last_change":"2024-04-22T08:27:47.962032+0000","last_active":"2024-04-22T08:27:53.380648+0000","last_peered":"2024-04-22T08:27:53.380648+0000","last_clean":"2024-04-22T08:27:53.380648+0000","last_became_active":"2024-04-22T08:27:47.912071+0000","last_became_peered":"2024-04-22T08:27:47.912071+0000","last_unstale":"2024-04-22T08:27:53.380648+0000","last_undegraded":"2024-04-22T08:27:53.380648+0000","last_fullsized":"2024-04-22T08:27:53.380648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:24.363832+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15712256,"num_objects":3836,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3836,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"32'3738","reported_seq":3887,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.365338+0000","last_change":"2024-04-22T08:27:52.931316+0000","last_active":"2024-04-22T08:27:53.365338+0000","last_peered":"2024-04-22T08:27:53.365338+0000","last_clean":"2024-04-22T08:27:53.365338+0000","last_became_active":"2024-04-22T08:27:47.865037+0000","last_became_peered":"2024-04-22T08:27:47.865037+0000","last_unstale":"2024-04-22T08:27:53.365338+0000","last_undegraded":"2024-04-22T08:27:53.365338+0000","last_fullsized":"2024-04-22T08:27:53.365338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:21:05.715899+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15286272,"num_objects":3732,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3732,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"32'3707","reported_seq":3845,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.371591+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:27:53.371591+0000","last_peered":"2024-04-22T08:27:53.371591+0000","last_clean":"2024-04-22T08:27:53.371591+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:53.371591+0000","last_undegraded":"2024-04-22T08:27:53.371591+0000","last_fullsized":"2024-04-22T08:27:53.371591+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:42:06.940931+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15175680,"num_objects":3705,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":1420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"32'3953","reported_seq":4064,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.082918+0000","last_change":"2024-04-22T08:27:54.083074+0000","last_active":"2024-04-22T08:27:54.082918+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.895678+0000","last_became_peered":"2024-04-22T08:27:47.895678+0000","last_unstale":"2024-04-22T08:27:54.082918+0000","last_undegraded":"2024-04-22T08:27:54.082918+0000","last_fullsized":"2024-04-22T08:27:54.082918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":3953,"log_dups_size":0,"ondisk_log_size":3953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15926482,"num_objects":3890,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":15556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.212030+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:51.212030+0000","last_peered":"2024-04-22T08:27:51.212030+0000","last_clean":"2024-04-22T08:27:51.212030+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:51.212030+0000","last_undegraded":"2024-04-22T08:27:51.212030+0000","last_fullsized":"2024-04-22T08:27:51.212030+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"32'3731","reported_seq":3795,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.330904+0000","last_change":"2024-04-22T08:27:47.975278+0000","last_active":"2024-04-22T08:27:53.330904+0000","last_peered":"2024-04-22T08:27:53.330904+0000","last_clean":"2024-04-22T08:27:53.330904+0000","last_became_active":"2024-04-22T08:27:47.907574+0000","last_became_peered":"2024-04-22T08:27:47.907574+0000","last_unstale":"2024-04-22T08:27:53.330904+0000","last_undegraded":"2024-04-22T08:27:53.330904+0000","last_fullsized":"2024-04-22T08:27:53.330904+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15250642,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":14896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.921458+0000","last_change":"2024-04-22T08:27:47.907718+0000","last_active":"2024-04-22T08:27:47.921458+0000","last_peered":"2024-04-22T08:27:47.921458+0000","last_clean":"2024-04-22T08:27:47.921458+0000","last_became_active":"2024-04-22T08:27:47.904280+0000","last_became_peered":"2024-04-22T08:27:47.904280+0000","last_unstale":"2024-04-22T08:27:47.921458+0000","last_undegraded":"2024-04-22T08:27:47.921458+0000","last_fullsized":"2024-04-22T08:27:47.921458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:54.701862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"32'3755","reported_seq":3889,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084354+0000","last_change":"2024-04-22T08:27:54.084505+0000","last_active":"2024-04-22T08:27:54.084354+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.891656+0000","last_became_peered":"2024-04-22T08:27:47.891656+0000","last_unstale":"2024-04-22T08:27:54.084354+0000","last_undegraded":"2024-04-22T08:27:54.084354+0000","last_fullsized":"2024-04-22T08:27:54.084354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:14:17.011512+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15360000,"num_objects":3750,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"32'3809","reported_seq":3852,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.083001+0000","last_change":"2024-04-22T08:27:54.083165+0000","last_active":"2024-04-22T08:27:54.083001+0000","last_peered":"2024-04-22T08:27:54.074517+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:47.897092+0000","last_became_peered":"2024-04-22T08:27:47.897092+0000","last_unstale":"2024-04-22T08:27:54.083001+0000","last_undegraded":"2024-04-22T08:27:54.083001+0000","last_fullsized":"2024-04-22T08:27:54.083001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15578322,"num_objects":3805,"num_object_clones":0,"num_object_copies":7610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":15215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"32'3673","reported_seq":3738,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.412759+0000","last_change":"2024-04-22T08:27:47.943170+0000","last_active":"2024-04-22T08:27:53.412759+0000","last_peered":"2024-04-22T08:27:53.412759+0000","last_clean":"2024-04-22T08:27:53.412759+0000","last_became_active":"2024-04-22T08:27:47.916592+0000","last_became_peered":"2024-04-22T08:27:47.916592+0000","last_unstale":"2024-04-22T08:27:53.412759+0000","last_undegraded":"2024-04-22T08:27:53.412759+0000","last_fullsized":"2024-04-22T08:27:53.412759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15037033,"num_objects":3672,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3672,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":14685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3672}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"32'3798","reported_seq":3930,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.405662+0000","last_change":"2024-04-22T08:27:51.614530+0000","last_active":"2024-04-22T08:27:53.405662+0000","last_peered":"2024-04-22T08:27:53.405662+0000","last_clean":"2024-04-22T08:27:53.405662+0000","last_became_active":"2024-04-22T08:27:47.910831+0000","last_became_peered":"2024-04-22T08:27:47.910831+0000","last_unstale":"2024-04-22T08:27:53.405662+0000","last_undegraded":"2024-04-22T08:27:53.405662+0000","last_fullsized":"2024-04-22T08:27:53.405662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2849,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15525691,"num_objects":3793,"num_object_clones":0,"num_object_copies":7586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3793,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":15164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.208360+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:51.208360+0000","last_peered":"2024-04-22T08:27:51.208360+0000","last_clean":"2024-04-22T08:27:51.208360+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:51.208360+0000","last_undegraded":"2024-04-22T08:27:51.208360+0000","last_fullsized":"2024-04-22T08:27:51.208360+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"32'3661","reported_seq":3723,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.395008+0000","last_change":"2024-04-22T08:27:47.884233+0000","last_active":"2024-04-22T08:27:53.395008+0000","last_peered":"2024-04-22T08:27:53.395008+0000","last_clean":"2024-04-22T08:27:53.395008+0000","last_became_active":"2024-04-22T08:27:47.864504+0000","last_became_peered":"2024-04-22T08:27:47.864504+0000","last_unstale":"2024-04-22T08:27:53.395008+0000","last_undegraded":"2024-04-22T08:27:53.395008+0000","last_fullsized":"2024-04-22T08:27:53.395008+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980306,"num_objects":3659,"num_object_clones":0,"num_object_copies":7318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":14630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"32'3838","reported_seq":3944,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.406331+0000","last_change":"2024-04-22T08:27:52.631343+0000","last_active":"2024-04-22T08:27:53.406331+0000","last_peered":"2024-04-22T08:27:53.406331+0000","last_clean":"2024-04-22T08:27:53.406331+0000","last_became_active":"2024-04-22T08:27:47.924447+0000","last_became_peered":"2024-04-22T08:27:47.924447+0000","last_unstale":"2024-04-22T08:27:53.406331+0000","last_undegraded":"2024-04-22T08:27:53.406331+0000","last_fullsized":"2024-04-22T08:27:53.406331+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":1599,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15673764,"num_objects":3830,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3830,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495394816,"data_stored":495307841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390700,"kb_used_data":242596,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93981140,"statfs":{"total":96636764160,"available":96236687360,"internally_reserved":0,"allocated":248418304,"data_stored":248231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920770,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":430080,"data_stored":152527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":32370380},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313234,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381480,"kb_used_data":237532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93990360,"statfs":{"total":96636764160,"available":96246128640,"internally_reserved":0,"allocated":243232768,"data_stored":243052862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246091776,"data_stored":246088297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":71535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249151488,"data_stored":249148009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:58.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:27:59.505 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:27:59.505 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:27:59.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-04-22T08:27:58.149986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":39,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":804236,"kb_used_data":480548,"kb_used_omap":12,"kb_used_meta":323635,"kb_avail":282311284,"statfs":{"total":289910292480,"available":289086754816,"internally_reserved":0,"allocated":492081152,"data_stored":491437325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":331402350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,7,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21513820,"num_objects":5249,"num_object_clones":0,"num_object_copies":10498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11253,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1921,"num_write_kb":-7663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001302"},"pg_stats":[{"pgid":"2.f","version":"32'3754","reported_seq":3807,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.390426+0000","last_change":"2024-04-22T08:27:47.905466+0000","last_active":"2024-04-22T08:27:53.390426+0000","last_peered":"2024-04-22T08:27:53.390426+0000","last_clean":"2024-04-22T08:27:53.390426+0000","last_became_active":"2024-04-22T08:27:47.898939+0000","last_became_peered":"2024-04-22T08:27:47.898939+0000","last_unstale":"2024-04-22T08:27:53.390426+0000","last_undegraded":"2024-04-22T08:27:53.390426+0000","last_fullsized":"2024-04-22T08:27:53.390426+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:27.633840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15376384,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":15016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3754}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"33'3819","reported_seq":3929,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.094077+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:54.094077+0000","last_peered":"2024-04-22T08:27:54.094077+0000","last_clean":"2024-04-22T08:27:54.094077+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:54.094077+0000","last_undegraded":"2024-04-22T08:27:54.094077+0000","last_fullsized":"2024-04-22T08:27:54.094077+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:19.957180+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":3500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"32'3759","reported_seq":3870,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084517+0000","last_change":"2024-04-22T08:27:54.084606+0000","last_active":"2024-04-22T08:27:54.084517+0000","last_peered":"2024-04-22T08:27:54.071080+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:47.865324+0000","last_became_peered":"2024-04-22T08:27:47.865324+0000","last_unstale":"2024-04-22T08:27:54.084517+0000","last_undegraded":"2024-04-22T08:27:54.084517+0000","last_fullsized":"2024-04-22T08:27:54.084517+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:08.298486+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15380480,"num_objects":3755,"num_object_clones":0,"num_object_copies":7510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":586,"num_write_kb":2344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"32'3888","reported_seq":3913,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:54.075301+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:54.075301+0000","last_peered":"2024-04-22T08:27:54.075301+0000","last_clean":"2024-04-22T08:27:54.075301+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:54.075301+0000","last_undegraded":"2024-04-22T08:27:54.075301+0000","last_fullsized":"2024-04-22T08:27:54.075301+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:17.879354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":3888,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"32'3770","reported_seq":3902,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.372138+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:27:53.372138+0000","last_peered":"2024-04-22T08:27:53.372138+0000","last_clean":"2024-04-22T08:27:53.372138+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:53.372138+0000","last_undegraded":"2024-04-22T08:27:53.372138+0000","last_fullsized":"2024-04-22T08:27:53.372138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:13.316934+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15413248,"num_objects":3763,"num_object_clones":0,"num_object_copies":7526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":3448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"32'3841","reported_seq":3989,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.380648+0000","last_change":"2024-04-22T08:27:47.962032+0000","last_active":"2024-04-22T08:27:53.380648+0000","last_peered":"2024-04-22T08:27:53.380648+0000","last_clean":"2024-04-22T08:27:53.380648+0000","last_became_active":"2024-04-22T08:27:47.912071+0000","last_became_peered":"2024-04-22T08:27:47.912071+0000","last_unstale":"2024-04-22T08:27:53.380648+0000","last_undegraded":"2024-04-22T08:27:53.380648+0000","last_fullsized":"2024-04-22T08:27:53.380648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:24.363832+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15712256,"num_objects":3836,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3836,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"32'3738","reported_seq":3887,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.365338+0000","last_change":"2024-04-22T08:27:52.931316+0000","last_active":"2024-04-22T08:27:53.365338+0000","last_peered":"2024-04-22T08:27:53.365338+0000","last_clean":"2024-04-22T08:27:53.365338+0000","last_became_active":"2024-04-22T08:27:47.865037+0000","last_became_peered":"2024-04-22T08:27:47.865037+0000","last_unstale":"2024-04-22T08:27:53.365338+0000","last_undegraded":"2024-04-22T08:27:53.365338+0000","last_fullsized":"2024-04-22T08:27:53.365338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:21:05.715899+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15286272,"num_objects":3732,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3732,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"32'3707","reported_seq":3845,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:53.371591+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:27:53.371591+0000","last_peered":"2024-04-22T08:27:53.371591+0000","last_clean":"2024-04-22T08:27:53.371591+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:53.371591+0000","last_undegraded":"2024-04-22T08:27:53.371591+0000","last_fullsized":"2024-04-22T08:27:53.371591+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:42:06.940931+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15175680,"num_objects":3705,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":1420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"32'3953","reported_seq":4064,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.082918+0000","last_change":"2024-04-22T08:27:54.083074+0000","last_active":"2024-04-22T08:27:54.082918+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.895678+0000","last_became_peered":"2024-04-22T08:27:47.895678+0000","last_unstale":"2024-04-22T08:27:54.082918+0000","last_undegraded":"2024-04-22T08:27:54.082918+0000","last_fullsized":"2024-04-22T08:27:54.082918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":3953,"log_dups_size":0,"ondisk_log_size":3953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15926482,"num_objects":3890,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":15556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.212030+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:51.212030+0000","last_peered":"2024-04-22T08:27:51.212030+0000","last_clean":"2024-04-22T08:27:51.212030+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:51.212030+0000","last_undegraded":"2024-04-22T08:27:51.212030+0000","last_fullsized":"2024-04-22T08:27:51.212030+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:08:13.375328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"32'3731","reported_seq":3795,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.330904+0000","last_change":"2024-04-22T08:27:47.975278+0000","last_active":"2024-04-22T08:27:53.330904+0000","last_peered":"2024-04-22T08:27:53.330904+0000","last_clean":"2024-04-22T08:27:53.330904+0000","last_became_active":"2024-04-22T08:27:47.907574+0000","last_became_peered":"2024-04-22T08:27:47.907574+0000","last_unstale":"2024-04-22T08:27:53.330904+0000","last_undegraded":"2024-04-22T08:27:53.330904+0000","last_fullsized":"2024-04-22T08:27:53.330904+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15250642,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":14896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:47.921458+0000","last_change":"2024-04-22T08:27:47.907718+0000","last_active":"2024-04-22T08:27:47.921458+0000","last_peered":"2024-04-22T08:27:47.921458+0000","last_clean":"2024-04-22T08:27:47.921458+0000","last_became_active":"2024-04-22T08:27:47.904280+0000","last_became_peered":"2024-04-22T08:27:47.904280+0000","last_unstale":"2024-04-22T08:27:47.921458+0000","last_undegraded":"2024-04-22T08:27:47.921458+0000","last_fullsized":"2024-04-22T08:27:47.921458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:50:54.701862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"32'3755","reported_seq":3889,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.084354+0000","last_change":"2024-04-22T08:27:54.084505+0000","last_active":"2024-04-22T08:27:54.084354+0000","last_peered":"2024-04-22T08:27:54.070471+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:47.891656+0000","last_became_peered":"2024-04-22T08:27:47.891656+0000","last_unstale":"2024-04-22T08:27:54.084354+0000","last_undegraded":"2024-04-22T08:27:54.084354+0000","last_fullsized":"2024-04-22T08:27:54.084354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:14:17.011512+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15360000,"num_objects":3750,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"32'3809","reported_seq":3852,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-04-22T08:27:54.083001+0000","last_change":"2024-04-22T08:27:54.083165+0000","last_active":"2024-04-22T08:27:54.083001+0000","last_peered":"2024-04-22T08:27:54.074517+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:47.897092+0000","last_became_peered":"2024-04-22T08:27:47.897092+0000","last_unstale":"2024-04-22T08:27:54.083001+0000","last_undegraded":"2024-04-22T08:27:54.083001+0000","last_fullsized":"2024-04-22T08:27:54.083001+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15578322,"num_objects":3805,"num_object_clones":0,"num_object_copies":7610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":15215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"32'3673","reported_seq":3738,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.412759+0000","last_change":"2024-04-22T08:27:47.943170+0000","last_active":"2024-04-22T08:27:53.412759+0000","last_peered":"2024-04-22T08:27:53.412759+0000","last_clean":"2024-04-22T08:27:53.412759+0000","last_became_active":"2024-04-22T08:27:47.916592+0000","last_became_peered":"2024-04-22T08:27:47.916592+0000","last_unstale":"2024-04-22T08:27:53.412759+0000","last_undegraded":"2024-04-22T08:27:53.412759+0000","last_fullsized":"2024-04-22T08:27:53.412759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15037033,"num_objects":3672,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3672,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":14685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3672}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"32'3798","reported_seq":3930,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.405662+0000","last_change":"2024-04-22T08:27:51.614530+0000","last_active":"2024-04-22T08:27:53.405662+0000","last_peered":"2024-04-22T08:27:53.405662+0000","last_clean":"2024-04-22T08:27:53.405662+0000","last_became_active":"2024-04-22T08:27:47.910831+0000","last_became_peered":"2024-04-22T08:27:47.910831+0000","last_unstale":"2024-04-22T08:27:53.405662+0000","last_undegraded":"2024-04-22T08:27:53.405662+0000","last_fullsized":"2024-04-22T08:27:53.405662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2849,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15525691,"num_objects":3793,"num_object_clones":0,"num_object_copies":7586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3793,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":15164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-22T08:27:51.208360+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:51.208360+0000","last_peered":"2024-04-22T08:27:51.208360+0000","last_clean":"2024-04-22T08:27:51.208360+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:51.208360+0000","last_undegraded":"2024-04-22T08:27:51.208360+0000","last_fullsized":"2024-04-22T08:27:51.208360+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:21.573250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"32'3661","reported_seq":3723,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2024-04-22T08:27:53.395008+0000","last_change":"2024-04-22T08:27:47.884233+0000","last_active":"2024-04-22T08:27:53.395008+0000","last_peered":"2024-04-22T08:27:53.395008+0000","last_clean":"2024-04-22T08:27:53.395008+0000","last_became_active":"2024-04-22T08:27:47.864504+0000","last_became_peered":"2024-04-22T08:27:47.864504+0000","last_unstale":"2024-04-22T08:27:53.395008+0000","last_undegraded":"2024-04-22T08:27:53.395008+0000","last_fullsized":"2024-04-22T08:27:53.395008+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980306,"num_objects":3659,"num_object_clones":0,"num_object_copies":7318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":14630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"32'3838","reported_seq":3944,"reported_epoch":32,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-04-22T08:27:53.406331+0000","last_change":"2024-04-22T08:27:52.631343+0000","last_active":"2024-04-22T08:27:53.406331+0000","last_peered":"2024-04-22T08:27:53.406331+0000","last_clean":"2024-04-22T08:27:53.406331+0000","last_became_active":"2024-04-22T08:27:47.924447+0000","last_became_peered":"2024-04-22T08:27:47.924447+0000","last_unstale":"2024-04-22T08:27:53.406331+0000","last_undegraded":"2024-04-22T08:27:53.406331+0000","last_fullsized":"2024-04-22T08:27:53.406331+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":1599,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15673764,"num_objects":3830,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3830,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":247219856,"num_objects":60370,"num_object_clones":0,"num_object_copies":120740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30001,"num_objects_unfound":0,"num_objects_dirty":60370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37735,"num_write_kb":150837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495394816,"data_stored":495307841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60494,"ondisk_log_size":60494,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390700,"kb_used_data":242596,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93981140,"statfs":{"total":96636764160,"available":96236687360,"internally_reserved":0,"allocated":248418304,"data_stored":248231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920770,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":430080,"data_stored":152527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":32370380},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313234,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381480,"kb_used_data":237532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93990360,"statfs":{"total":96636764160,"available":96246128640,"internally_reserved":0,"allocated":243232768,"data_stored":243052862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246091776,"data_stored":246088297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":71535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249151488,"data_stored":249148009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:27:59.548 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:27:59.548 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "32'3754", 'reported_seq': 3807, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.390426+0000', 'last_change': '2024-04-22T08:27:47.905466+0000', 'last_active': '2024-04-22T08:27:53.390426+0000', 'last_peered': '2024-04-22T08:27:53.390426+0000', 'last_clean': '2024-04-22T08:27:53.390426+0000', 'last_became_active': '2024-04-22T08:27:47.898939+0000', 'last_became_peered': '2024-04-22T08:27:47.898939+0000', 'last_unstale': '2024-04-22T08:27:53.390426+0000', 'last_undegraded': '2024-04-22T08:27:53.390426+0000', 'last_fullsized': '2024-04-22T08:27:53.390426+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3754, 'log_dups_size': 0, 'ondisk_log_size': 3754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:17:27.633840+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15376384, 'num_objects': 3754, 'num_object_clones': 0, 'num_object_copies': 7508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3754, 'num_objects_unfound': 0, 'num_objects_dirty': 3754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3754, 'num_write_kb': 15016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3754}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.548 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:27:59.548 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "32'3759", 'reported_seq': 3870, 'reported_epoch': 33, 'state': 'remapped+peering', 'last_fresh': '2024-04-22T08:27:54.084517+0000', 'last_change': '2024-04-22T08:27:54.084606+0000', 'last_active': '2024-04-22T08:27:54.084517+0000', 'last_peered': '2024-04-22T08:27:54.071080+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:47.865324+0000', 'last_became_peered': '2024-04-22T08:27:47.865324+0000', 'last_unstale': '2024-04-22T08:27:54.084517+0000', 'last_undegraded': '2024-04-22T08:27:54.084517+0000', 'last_fullsized': '2024-04-22T08:27:54.084517+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 3759, 'log_dups_size': 0, 'ondisk_log_size': 3759, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T16:18:08.298486+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15380480, 'num_objects': 3755, 'num_object_clones': 0, 'num_object_copies': 7510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 586, 'num_write_kb': 2344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "32'3841", 'reported_seq': 3989, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.380648+0000', 'last_change': '2024-04-22T08:27:47.962032+0000', 'last_active': '2024-04-22T08:27:53.380648+0000', 'last_peered': '2024-04-22T08:27:53.380648+0000', 'last_clean': '2024-04-22T08:27:53.380648+0000', 'last_became_active': '2024-04-22T08:27:47.912071+0000', 'last_became_peered': '2024-04-22T08:27:47.912071+0000', 'last_unstale': '2024-04-22T08:27:53.380648+0000', 'last_undegraded': '2024-04-22T08:27:53.380648+0000', 'last_fullsized': '2024-04-22T08:27:53.380648+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 3841, 'log_dups_size': 0, 'ondisk_log_size': 3841, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T09:25:24.363832+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15712256, 'num_objects': 3836, 'num_object_clones': 0, 'num_object_copies': 7672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3836, 'num_objects_unfound': 0, 'num_objects_dirty': 3836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 710, 'num_write_kb': 2840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3836}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "32'3738", 'reported_seq': 3887, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.365338+0000', 'last_change': '2024-04-22T08:27:52.931316+0000', 'last_active': '2024-04-22T08:27:53.365338+0000', 'last_peered': '2024-04-22T08:27:53.365338+0000', 'last_clean': '2024-04-22T08:27:53.365338+0000', 'last_became_active': '2024-04-22T08:27:47.865037+0000', 'last_became_peered': '2024-04-22T08:27:47.865037+0000', 'last_unstale': '2024-04-22T08:27:53.365338+0000', 'last_undegraded': '2024-04-22T08:27:53.365338+0000', 'last_fullsized': '2024-04-22T08:27:53.365338+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 3738, 'log_dups_size': 0, 'ondisk_log_size': 3738, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T09:21:05.715899+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15286272, 'num_objects': 3732, 'num_object_clones': 0, 'num_object_copies': 7464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3732, 'num_objects_unfound': 0, 'num_objects_dirty': 3732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 106, 'num_write_kb': 424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3732}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "32'3953", 'reported_seq': 4064, 'reported_epoch': 33, 'state': 'remapped+peering', 'last_fresh': '2024-04-22T08:27:54.082918+0000', 'last_change': '2024-04-22T08:27:54.083074+0000', 'last_active': '2024-04-22T08:27:54.082918+0000', 'last_peered': '2024-04-22T08:27:54.070471+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:47.895678+0000', 'last_became_peered': '2024-04-22T08:27:47.895678+0000', 'last_unstale': '2024-04-22T08:27:54.082918+0000', 'last_undegraded': '2024-04-22T08:27:54.082918+0000', 'last_fullsized': '2024-04-22T08:27:54.082918+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 3953, 'log_dups_size': 0, 'ondisk_log_size': 3953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15926482, 'num_objects': 3890, 'num_object_clones': 0, 'num_object_copies': 7780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3893, 'num_write_kb': 15556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "32'3731", 'reported_seq': 3795, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.330904+0000', 'last_change': '2024-04-22T08:27:47.975278+0000', 'last_active': '2024-04-22T08:27:53.330904+0000', 'last_peered': '2024-04-22T08:27:53.330904+0000', 'last_clean': '2024-04-22T08:27:53.330904+0000', 'last_became_active': '2024-04-22T08:27:47.907574+0000', 'last_became_peered': '2024-04-22T08:27:47.907574+0000', 'last_unstale': '2024-04-22T08:27:53.330904+0000', 'last_undegraded': '2024-04-22T08:27:53.330904+0000', 'last_fullsized': '2024-04-22T08:27:53.330904+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3731, 'log_dups_size': 0, 'ondisk_log_size': 3731, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15250642, 'num_objects': 3725, 'num_object_clones': 0, 'num_object_copies': 7450, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3725, 'num_objects_unfound': 0, 'num_objects_dirty': 3725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3729, 'num_write_kb': 14896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3725}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 1.2 is not active+clean 2024-04-22T08:27:59.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '1.2', 'version': "0'0", 'reported_seq': 52, 'reported_epoch': 31, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:47.921458+0000', 'last_change': '2024-04-22T08:27:47.907718+0000', 'last_active': '2024-04-22T08:27:47.921458+0000', 'last_peered': '2024-04-22T08:27:47.921458+0000', 'last_clean': '2024-04-22T08:27:47.921458+0000', 'last_became_active': '2024-04-22T08:27:47.904280+0000', 'last_became_peered': '2024-04-22T08:27:47.904280+0000', 'last_unstale': '2024-04-22T08:27:47.921458+0000', 'last_undegraded': '2024-04-22T08:27:47.921458+0000', 'last_fullsized': '2024-04-22T08:27:47.921458+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:14.194929+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:14.194929+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:14.194929+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:50:54.701862+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "32'3755", 'reported_seq': 3889, 'reported_epoch': 33, 'state': 'remapped+peering', 'last_fresh': '2024-04-22T08:27:54.084354+0000', 'last_change': '2024-04-22T08:27:54.084505+0000', 'last_active': '2024-04-22T08:27:54.084354+0000', 'last_peered': '2024-04-22T08:27:54.070471+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:47.891656+0000', 'last_became_peered': '2024-04-22T08:27:47.891656+0000', 'last_unstale': '2024-04-22T08:27:54.084354+0000', 'last_undegraded': '2024-04-22T08:27:54.084354+0000', 'last_fullsized': '2024-04-22T08:27:54.084354+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 3755, 'log_dups_size': 0, 'ondisk_log_size': 3755, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T16:14:17.011512+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15360000, 'num_objects': 3750, 'num_object_clones': 0, 'num_object_copies': 7500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 209, 'num_write_kb': 836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "32'3809", 'reported_seq': 3852, 'reported_epoch': 33, 'state': 'remapped+peering', 'last_fresh': '2024-04-22T08:27:54.083001+0000', 'last_change': '2024-04-22T08:27:54.083165+0000', 'last_active': '2024-04-22T08:27:54.083001+0000', 'last_peered': '2024-04-22T08:27:54.074517+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:47.897092+0000', 'last_became_peered': '2024-04-22T08:27:47.897092+0000', 'last_unstale': '2024-04-22T08:27:54.083001+0000', 'last_undegraded': '2024-04-22T08:27:54.083001+0000', 'last_fullsized': '2024-04-22T08:27:54.083001+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3809, 'log_dups_size': 0, 'ondisk_log_size': 3809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15578322, 'num_objects': 3805, 'num_object_clones': 0, 'num_object_copies': 7610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3807, 'num_write_kb': 15215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "32'3673", 'reported_seq': 3738, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.412759+0000', 'last_change': '2024-04-22T08:27:47.943170+0000', 'last_active': '2024-04-22T08:27:53.412759+0000', 'last_peered': '2024-04-22T08:27:53.412759+0000', 'last_clean': '2024-04-22T08:27:53.412759+0000', 'last_became_active': '2024-04-22T08:27:47.916592+0000', 'last_became_peered': '2024-04-22T08:27:47.916592+0000', 'last_unstale': '2024-04-22T08:27:53.412759+0000', 'last_undegraded': '2024-04-22T08:27:53.412759+0000', 'last_fullsized': '2024-04-22T08:27:53.412759+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3673, 'log_dups_size': 0, 'ondisk_log_size': 3673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15037033, 'num_objects': 3672, 'num_object_clones': 0, 'num_object_copies': 7344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3672, 'num_objects_unfound': 0, 'num_objects_dirty': 3672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3672, 'num_write_kb': 14685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3672}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "32'3798", 'reported_seq': 3930, 'reported_epoch': 32, 'state': 'active+clean+remapped+scrubbing', 'last_fresh': '2024-04-22T08:27:53.405662+0000', 'last_change': '2024-04-22T08:27:51.614530+0000', 'last_active': '2024-04-22T08:27:53.405662+0000', 'last_peered': '2024-04-22T08:27:53.405662+0000', 'last_clean': '2024-04-22T08:27:53.405662+0000', 'last_became_active': '2024-04-22T08:27:47.910831+0000', 'last_became_peered': '2024-04-22T08:27:47.910831+0000', 'last_unstale': '2024-04-22T08:27:53.405662+0000', 'last_undegraded': '2024-04-22T08:27:53.405662+0000', 'last_fullsized': '2024-04-22T08:27:53.405662+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2849, 'log_size': 3798, 'log_dups_size': 0, 'ondisk_log_size': 3798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15525691, 'num_objects': 3793, 'num_object_clones': 0, 'num_object_copies': 7586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3793, 'num_objects_unfound': 0, 'num_objects_dirty': 3793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3795, 'num_write_kb': 15164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3793}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:27:59.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "32'3661", 'reported_seq': 3723, 'reported_epoch': 32, 'state': 'active+clean+remapped', 'last_fresh': '2024-04-22T08:27:53.395008+0000', 'last_change': '2024-04-22T08:27:47.884233+0000', 'last_active': '2024-04-22T08:27:53.395008+0000', 'last_peered': '2024-04-22T08:27:53.395008+0000', 'last_clean': '2024-04-22T08:27:53.395008+0000', 'last_became_active': '2024-04-22T08:27:47.864504+0000', 'last_became_peered': '2024-04-22T08:27:47.864504+0000', 'last_unstale': '2024-04-22T08:27:53.395008+0000', 'last_undegraded': '2024-04-22T08:27:53.395008+0000', 'last_fullsized': '2024-04-22T08:27:53.395008+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3661, 'log_dups_size': 0, 'ondisk_log_size': 3661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14980306, 'num_objects': 3659, 'num_object_clones': 0, 'num_object_copies': 7318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3659, 'num_objects_unfound': 0, 'num_objects_dirty': 3659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3659, 'num_write_kb': 14630, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3659}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.551 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:27:59.551 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "32'3838", 'reported_seq': 3944, 'reported_epoch': 32, 'state': 'active+clean+remapped+scrubbing', 'last_fresh': '2024-04-22T08:27:53.406331+0000', 'last_change': '2024-04-22T08:27:52.631343+0000', 'last_active': '2024-04-22T08:27:53.406331+0000', 'last_peered': '2024-04-22T08:27:53.406331+0000', 'last_clean': '2024-04-22T08:27:53.406331+0000', 'last_became_active': '2024-04-22T08:27:47.924447+0000', 'last_became_peered': '2024-04-22T08:27:47.924447+0000', 'last_unstale': '2024-04-22T08:27:53.406331+0000', 'last_undegraded': '2024-04-22T08:27:53.406331+0000', 'last_fullsized': '2024-04-22T08:27:53.406331+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 1599, 'log_size': 3838, 'log_dups_size': 0, 'ondisk_log_size': 3838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15673764, 'num_objects': 3830, 'num_object_clones': 0, 'num_object_copies': 7660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3830, 'num_objects_unfound': 0, 'num_objects_dirty': 3830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3835, 'num_write_kb': 15311, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 3830}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:27:59.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:27:59.820 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 47 20 62762 62742 5.21356 2.37109 0.0309485 0.0149759 2024-04-22T08:28:00.520 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:00.520 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":115,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+clean+remapped","count":7},{"state_name":"remapped+peering","count":4},{"state_name":"active+clean+remapped+scrubbing","count":2}],"num_pgs":24,"num_pools":2,"num_objects":60370,"data_bytes":247219856,"bytes_used":823537664,"bytes_avail":289086754816,"bytes_total":289910292480,"inactive_pgs_ratio":0.1666666716337204,"misplaced_objects":30001,"misplaced_total":120740,"misplaced_ratio":0.24847606427033295},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:00.579 INFO:tasks.rep_lost_unfound_delete.ceph_manager:no progress seen, keeping timeout for now 2024-04-22T08:28:00.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:00.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 48 20 63384 63364 5.1554 2.42969 0.0241673 0.0151443 2024-04-22T08:28:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:01.320 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-04-22T08:28:00.150645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":254521173,"num_objects":62150,"num_object_clones":0,"num_object_copies":124300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3744,"num_objects_unfound":0,"num_objects_dirty":62150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35752,"num_write_kb":142921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":168553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62277,"ondisk_log_size":62277,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":795220,"kb_used_data":492016,"kb_used_omap":12,"kb_used_meta":303155,"kb_avail":282320300,"statfs":{"total":289910292480,"available":289095987200,"internally_reserved":0,"allocated":503824384,"data_stored":503151263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":310430830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,7,0,0,0,0,0,0,17],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14869491,"num_objects":3626,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":-40241,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3679,"num_write_kb":-14691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":143360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.078052"},"pg_stats":[{"pgid":"2.f","version":"35'3866","reported_seq":3926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.390840+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:27:58.390840+0000","last_peered":"2024-04-22T08:27:58.390840+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:27:58.390840+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":3866,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":15464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":72},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'3941","reported_seq":4053,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.472793+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:59.472793+0000","last_peered":"2024-04-22T08:27:59.472793+0000","last_clean":"2024-04-22T08:27:59.472793+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:59.472793+0000","last_undegraded":"2024-04-22T08:27:59.472793+0000","last_fullsized":"2024-04-22T08:27:59.472793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":997,"num_write_kb":3988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'3864","reported_seq":3984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.449400+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:27:59.449400+0000","last_peered":"2024-04-22T08:27:59.449400+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:27:59.449400+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15810560,"num_objects":3860,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":691,"num_write_kb":2764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'3989","reported_seq":4016,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.469673+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:59.469673+0000","last_peered":"2024-04-22T08:27:59.469673+0000","last_clean":"2024-04-22T08:27:59.469673+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:59.469673+0000","last_undegraded":"2024-04-22T08:27:59.469673+0000","last_fullsized":"2024-04-22T08:27:59.469673+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338944,"num_objects":3989,"num_object_clones":0,"num_object_copies":7978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":15956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'3870","reported_seq":4005,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:58.410275+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:27:58.410275+0000","last_peered":"2024-04-22T08:27:58.410275+0000","last_clean":"2024-04-22T08:27:58.410275+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:27:58.410275+0000","last_undegraded":"2024-04-22T08:27:58.410275+0000","last_fullsized":"2024-04-22T08:27:58.410275+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":3870,"log_dups_size":0,"ondisk_log_size":3870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15822848,"num_objects":3863,"num_object_clones":0,"num_object_copies":7726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":3848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'3965","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.403106+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:27:58.403106+0000","last_peered":"2024-04-22T08:27:58.403106+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:27:58.403106+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16220160,"num_objects":3960,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":834,"num_write_kb":3336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":81},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'3851","reported_seq":4084,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:27:58.426905+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:27:58.426905+0000","last_peered":"2024-04-22T08:27:58.426905+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:27:58.426905+0000","last_undegraded":"2024-04-22T08:27:58.426905+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15749120,"num_objects":3845,"num_object_clones":0,"num_object_copies":7690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3744,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":151552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":101},{"shards":"0,2","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'3823","reported_seq":3963,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:58.433480+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:27:58.433480+0000","last_peered":"2024-04-22T08:27:58.433480+0000","last_clean":"2024-04-22T08:27:58.433480+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:27:58.433480+0000","last_undegraded":"2024-04-22T08:27:58.433480+0000","last_fullsized":"2024-04-22T08:27:58.433480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15650816,"num_objects":3821,"num_object_clones":0,"num_object_copies":7642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":471,"num_write_kb":1884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4062","reported_seq":4182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.396751+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:27:59.396751+0000","last_peered":"2024-04-22T08:27:59.396751+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:27:59.396751+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372946,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":15992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'3845","reported_seq":3919,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.362421+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:27:58.362421+0000","last_peered":"2024-04-22T08:27:58.362421+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:27:58.362421+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15717586,"num_objects":3839,"num_object_clones":0,"num_object_copies":7678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":15352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":68},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'3862","reported_seq":4005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.251957+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:27:59.251957+0000","last_peered":"2024-04-22T08:27:59.251957+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:27:59.251957+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15798272,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'3916","reported_seq":3968,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.421463+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:27:59.421463+0000","last_peered":"2024-04-22T08:27:59.421463+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:27:59.421463+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16016594,"num_objects":3912,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":15643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'3796","reported_seq":3872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.356097+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:27:58.356097+0000","last_peered":"2024-04-22T08:27:58.356097+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:27:58.356097+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15540841,"num_objects":3795,"num_object_clones":0,"num_object_copies":7590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":15177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":74},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'3897","reported_seq":4069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.427709+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:27:58.427709+0000","last_peered":"2024-04-22T08:27:58.427709+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:27:58.427709+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15929344,"num_objects":3889,"num_object_clones":0,"num_object_copies":7778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":52},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'3772","reported_seq":3845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.402818+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:27:58.402818+0000","last_peered":"2024-04-22T08:27:58.402818+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:27:58.402818+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15434962,"num_objects":3770,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":15074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":65},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'3958","reported_seq":4099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:58.342497+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:27:58.342497+0000","last_peered":"2024-04-22T08:27:58.342497+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:27:58.342497+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16165284,"num_objects":3950,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3955,"num_write_kb":15791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":69},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":254521173,"num_objects":62150,"num_object_clones":0,"num_object_copies":124300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3744,"num_objects_unfound":0,"num_objects_dirty":62150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35752,"num_write_kb":142921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":168553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":505335808,"data_stored":505248833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62277,"ondisk_log_size":62277,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382244,"kb_used_data":246428,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93989596,"statfs":{"total":96636764160,"available":96245346304,"internally_reserved":0,"allocated":252342272,"data_stored":252141257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920770,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":430080,"data_stored":152527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":32370380},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313235,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380920,"kb_used_data":245168,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93990920,"statfs":{"total":96636764160,"available":96246702080,"internally_reserved":0,"allocated":251052032,"data_stored":250857479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":138997459},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252452864,"data_stored":252449385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":71535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252731392,"data_stored":252727913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:01.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 49 20 64013 63993 5.10033 2.45703 0.0817246 0.0153072 2024-04-22T08:28:02.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 50 20 64673 64653 5.04988 2.57812 0.0216291 0.0154595 2024-04-22T08:28:03.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 51 20 65344 65324 5.00226 2.62109 0.0658694 0.0156075 2024-04-22T08:28:04.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:04.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 52 20 65983 65963 4.95407 2.49609 0.0296995 0.015759 2024-04-22T08:28:05.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:05.276 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:05.310 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-04-22T08:28:04.151590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262750037,"num_objects":64159,"num_object_clones":0,"num_object_copies":128318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":64159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37761,"num_write_kb":150957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64286,"ondisk_log_size":64286,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":866796,"kb_used_data":511124,"kb_used_omap":12,"kb_used_meta":355635,"kb_avail":282248724,"statfs":{"total":289910292480,"available":289022693376,"internally_reserved":0,"allocated":523390976,"data_stored":522717855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":364170350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,3,5,14],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15530181,"num_objects":3789,"num_object_clones":0,"num_object_copies":7578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":-26369,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":606208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.074022"},"pg_stats":[{"pgid":"2.f","version":"35'4074","reported_seq":4134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.378441+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:03.378441+0000","last_peered":"2024-04-22T08:28:03.378441+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:03.378441+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":4074,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":16296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'3941","reported_seq":4053,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.472793+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:59.472793+0000","last_peered":"2024-04-22T08:27:59.472793+0000","last_clean":"2024-04-22T08:27:59.472793+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:59.472793+0000","last_undegraded":"2024-04-22T08:27:59.472793+0000","last_fullsized":"2024-04-22T08:27:59.472793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":997,"num_write_kb":3988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'3864","reported_seq":3984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.449400+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:27:59.449400+0000","last_peered":"2024-04-22T08:27:59.449400+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:27:59.449400+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15810560,"num_objects":3860,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":691,"num_write_kb":2764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'3989","reported_seq":4016,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.469673+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:59.469673+0000","last_peered":"2024-04-22T08:27:59.469673+0000","last_clean":"2024-04-22T08:27:59.469673+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:59.469673+0000","last_undegraded":"2024-04-22T08:27:59.469673+0000","last_fullsized":"2024-04-22T08:27:59.469673+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338944,"num_objects":3989,"num_object_clones":0,"num_object_copies":7978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":15956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4069","reported_seq":4204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.396436+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:03.396436+0000","last_peered":"2024-04-22T08:28:03.396436+0000","last_clean":"2024-04-22T08:28:03.396436+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:03.396436+0000","last_undegraded":"2024-04-22T08:28:03.396436+0000","last_fullsized":"2024-04-22T08:28:03.396436+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16637952,"num_objects":4062,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":4644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4159","reported_seq":4318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.459788+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:03.459788+0000","last_peered":"2024-04-22T08:28:03.459788+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:03.459788+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17014784,"num_objects":4154,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4047","reported_seq":4503,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:03.425604+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:03.425604+0000","last_peered":"2024-04-22T08:28:03.425604+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:03.425604+0000","last_undegraded":"2024-04-22T08:28:03.425604+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":409},{"shards":"0,2","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4021","reported_seq":4162,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.433626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:03.433626+0000","last_peered":"2024-04-22T08:28:03.433626+0000","last_clean":"2024-04-22T08:28:03.433626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:03.433626+0000","last_undegraded":"2024-04-22T08:28:03.433626+0000","last_fullsized":"2024-04-22T08:28:03.433626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4062","reported_seq":4182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.396751+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:27:59.396751+0000","last_peered":"2024-04-22T08:27:59.396751+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:27:59.396751+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372946,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":15992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4032","reported_seq":4106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.388967+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:03.388967+0000","last_peered":"2024-04-22T08:28:03.388967+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:03.388967+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16483538,"num_objects":4026,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":16100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":255},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'3862","reported_seq":4005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.251957+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:27:59.251957+0000","last_peered":"2024-04-22T08:27:59.251957+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:27:59.251957+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15798272,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'3916","reported_seq":3968,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.421463+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:27:59.421463+0000","last_peered":"2024-04-22T08:27:59.421463+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:27:59.421463+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16016594,"num_objects":3912,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":15643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4009","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.428369+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:03.428369+0000","last_peered":"2024-04-22T08:28:03.428369+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:03.428369+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413289,"num_objects":4008,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":16029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4106","reported_seq":4276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.425177+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:03.425177+0000","last_peered":"2024-04-22T08:28:03.425177+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:03.425177+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":1348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":261},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'3984","reported_seq":4055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.431849+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:03.431849+0000","last_peered":"2024-04-22T08:28:03.431849+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:03.431849+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303314,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":15922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":277},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4151","reported_seq":4292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.438583+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:03.438583+0000","last_peered":"2024-04-22T08:28:03.438583+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:03.438583+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16955812,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":262},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262750037,"num_objects":64159,"num_object_clones":0,"num_object_copies":128318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":64159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37761,"num_write_kb":150957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525156352,"data_stored":525069377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64286,"ondisk_log_size":64286,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382244,"kb_used_data":246428,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93989596,"statfs":{"total":96636764160,"available":96245346304,"internally_reserved":0,"allocated":252342272,"data_stored":252141257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920771,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85948,"kb_used_data":10036,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94285892,"statfs":{"total":96636764160,"available":96548753408,"internally_reserved":0,"allocated":10276864,"data_stored":9999311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":77721292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313236,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398604,"kb_used_data":254660,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93973236,"statfs":{"total":96636764160,"available":96228593664,"internally_reserved":0,"allocated":260771840,"data_stored":260577287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262242304,"data_stored":262238825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10182656,"data_stored":10102639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252731392,"data_stored":252727913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:05.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:05.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 53 20 66581 66561 4.90467 2.33594 0.0374648 0.0159191 2024-04-22T08:28:06.139 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:06.139 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:06.162 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-04-22T08:28:04.151590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262750037,"num_objects":64159,"num_object_clones":0,"num_object_copies":128318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":64159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37761,"num_write_kb":150957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64286,"ondisk_log_size":64286,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":866796,"kb_used_data":511124,"kb_used_omap":12,"kb_used_meta":355635,"kb_avail":282248724,"statfs":{"total":289910292480,"available":289022693376,"internally_reserved":0,"allocated":523390976,"data_stored":522717855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":364170350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,3,5,14],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15530181,"num_objects":3789,"num_object_clones":0,"num_object_copies":7578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":-26369,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":606208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.074022"},"pg_stats":[{"pgid":"2.f","version":"35'4074","reported_seq":4134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.378441+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:03.378441+0000","last_peered":"2024-04-22T08:28:03.378441+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:03.378441+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":4074,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":16296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'3941","reported_seq":4053,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.472793+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:27:59.472793+0000","last_peered":"2024-04-22T08:27:59.472793+0000","last_clean":"2024-04-22T08:27:59.472793+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:27:59.472793+0000","last_undegraded":"2024-04-22T08:27:59.472793+0000","last_fullsized":"2024-04-22T08:27:59.472793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":997,"num_write_kb":3988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'3864","reported_seq":3984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.449400+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:27:59.449400+0000","last_peered":"2024-04-22T08:27:59.449400+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:27:59.449400+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15810560,"num_objects":3860,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":691,"num_write_kb":2764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'3989","reported_seq":4016,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:59.469673+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:27:59.469673+0000","last_peered":"2024-04-22T08:27:59.469673+0000","last_clean":"2024-04-22T08:27:59.469673+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:27:59.469673+0000","last_undegraded":"2024-04-22T08:27:59.469673+0000","last_fullsized":"2024-04-22T08:27:59.469673+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338944,"num_objects":3989,"num_object_clones":0,"num_object_copies":7978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":15956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4069","reported_seq":4204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.396436+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:03.396436+0000","last_peered":"2024-04-22T08:28:03.396436+0000","last_clean":"2024-04-22T08:28:03.396436+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:03.396436+0000","last_undegraded":"2024-04-22T08:28:03.396436+0000","last_fullsized":"2024-04-22T08:28:03.396436+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16637952,"num_objects":4062,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":4644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4159","reported_seq":4318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.459788+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:03.459788+0000","last_peered":"2024-04-22T08:28:03.459788+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:03.459788+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17014784,"num_objects":4154,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4047","reported_seq":4503,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:03.425604+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:03.425604+0000","last_peered":"2024-04-22T08:28:03.425604+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:03.425604+0000","last_undegraded":"2024-04-22T08:28:03.425604+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":409},{"shards":"0,2","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4021","reported_seq":4162,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.433626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:03.433626+0000","last_peered":"2024-04-22T08:28:03.433626+0000","last_clean":"2024-04-22T08:28:03.433626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:03.433626+0000","last_undegraded":"2024-04-22T08:28:03.433626+0000","last_fullsized":"2024-04-22T08:28:03.433626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4062","reported_seq":4182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.396751+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:27:59.396751+0000","last_peered":"2024-04-22T08:27:59.396751+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:27:59.396751+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16372946,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4002,"num_write_kb":15992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4032","reported_seq":4106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.388967+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:03.388967+0000","last_peered":"2024-04-22T08:28:03.388967+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:03.388967+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16483538,"num_objects":4026,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":16100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":255},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'3862","reported_seq":4005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.251957+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:27:59.251957+0000","last_peered":"2024-04-22T08:27:59.251957+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:27:59.251957+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15798272,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'3916","reported_seq":3968,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:27:59.421463+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:27:59.421463+0000","last_peered":"2024-04-22T08:27:59.421463+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:27:59.421463+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16016594,"num_objects":3912,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":15643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4009","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.428369+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:03.428369+0000","last_peered":"2024-04-22T08:28:03.428369+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:03.428369+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413289,"num_objects":4008,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":16029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4106","reported_seq":4276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.425177+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:03.425177+0000","last_peered":"2024-04-22T08:28:03.425177+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:03.425177+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":1348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":261},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'3984","reported_seq":4055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.431849+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:03.431849+0000","last_peered":"2024-04-22T08:28:03.431849+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:03.431849+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303314,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":15922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":277},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4151","reported_seq":4292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.438583+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:03.438583+0000","last_peered":"2024-04-22T08:28:03.438583+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:03.438583+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16955812,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":262},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262750037,"num_objects":64159,"num_object_clones":0,"num_object_copies":128318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":64159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37761,"num_write_kb":150957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525156352,"data_stored":525069377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64286,"ondisk_log_size":64286,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382244,"kb_used_data":246428,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93989596,"statfs":{"total":96636764160,"available":96245346304,"internally_reserved":0,"allocated":252342272,"data_stored":252141257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920771,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85948,"kb_used_data":10036,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94285892,"statfs":{"total":96636764160,"available":96548753408,"internally_reserved":0,"allocated":10276864,"data_stored":9999311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":77721292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313236,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398604,"kb_used_data":254660,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93973236,"statfs":{"total":96636764160,"available":96228593664,"internally_reserved":0,"allocated":260771840,"data_stored":260577287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262242304,"data_stored":262238825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10182656,"data_stored":10102639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252731392,"data_stored":252727913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'4074", 'reported_seq': 4134, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.378441+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:03.378441+0000', 'last_peered': '2024-04-22T08:28:03.378441+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:03.378441+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4074, 'log_dups_size': 0, 'ondisk_log_size': 4074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16687104, 'num_objects': 4074, 'num_object_clones': 0, 'num_object_copies': 8148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4074, 'num_write_kb': 16296, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 280}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'3864", 'reported_seq': 3984, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:27:59.449400+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:27:59.449400+0000', 'last_peered': '2024-04-22T08:27:59.449400+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:27:59.449400+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 3864, 'log_dups_size': 0, 'ondisk_log_size': 3864, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15810560, 'num_objects': 3860, 'num_object_clones': 0, 'num_object_copies': 7720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 691, 'num_write_kb': 2764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:06.163 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'4159", 'reported_seq': 4318, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.459788+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:03.459788+0000', 'last_peered': '2024-04-22T08:28:03.459788+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:03.459788+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 4159, 'log_dups_size': 0, 'ondisk_log_size': 4159, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17014784, 'num_objects': 4154, 'num_object_clones': 0, 'num_object_copies': 8308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1028, 'num_write_kb': 4112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 275}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'4047", 'reported_seq': 4503, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:03.425604+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:03.425604+0000', 'last_peered': '2024-04-22T08:28:03.425604+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:03.425604+0000', 'last_undegraded': '2024-04-22T08:28:03.425604+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 4047, 'log_dups_size': 0, 'ondisk_log_size': 4047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16551936, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 8082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3632, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 415, 'num_write_kb': 1660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 149, 'num_bytes_recovered': 610304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 409}, {'shards': '0,2', 'objects': 3632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'4062", 'reported_seq': 4182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:27:59.396751+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:27:59.396751+0000', 'last_peered': '2024-04-22T08:27:59.396751+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:27:59.396751+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 4062, 'log_dups_size': 0, 'ondisk_log_size': 4062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16372946, 'num_objects': 3999, 'num_object_clones': 0, 'num_object_copies': 7998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4002, 'num_write_kb': 15992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'4032", 'reported_seq': 4106, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.388967+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:03.388967+0000', 'last_peered': '2024-04-22T08:28:03.388967+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:03.388967+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4032, 'log_dups_size': 0, 'ondisk_log_size': 4032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16483538, 'num_objects': 4026, 'num_object_clones': 0, 'num_object_copies': 8052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4030, 'num_write_kb': 16100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 255}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'3862", 'reported_seq': 4005, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:27:59.251957+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:27:59.251957+0000', 'last_peered': '2024-04-22T08:27:59.251957+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:27:59.251957+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 3862, 'log_dups_size': 0, 'ondisk_log_size': 3862, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15798272, 'num_objects': 3857, 'num_object_clones': 0, 'num_object_copies': 7714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 316, 'num_write_kb': 1264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 107}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:06.164 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'3916", 'reported_seq': 3968, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:27:59.421463+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:27:59.421463+0000', 'last_peered': '2024-04-22T08:27:59.421463+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:27:59.421463+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3916, 'log_dups_size': 0, 'ondisk_log_size': 3916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16016594, 'num_objects': 3912, 'num_object_clones': 0, 'num_object_copies': 7824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3914, 'num_write_kb': 15643, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'4009", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.428369+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:03.428369+0000', 'last_peered': '2024-04-22T08:28:03.428369+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:03.428369+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4009, 'log_dups_size': 0, 'ondisk_log_size': 4009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16413289, 'num_objects': 4008, 'num_object_clones': 0, 'num_object_copies': 8016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4008, 'num_write_kb': 16029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 287}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'4106", 'reported_seq': 4276, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.425177+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:03.425177+0000', 'last_peered': '2024-04-22T08:28:03.425177+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:03.425177+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 4106, 'log_dups_size': 0, 'ondisk_log_size': 4106, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16785408, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 8196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 337, 'num_write_kb': 1348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 261}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'3984", 'reported_seq': 4055, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.431849+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:03.431849+0000', 'last_peered': '2024-04-22T08:28:03.431849+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:03.431849+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 3984, 'log_dups_size': 0, 'ondisk_log_size': 3984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16303314, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 7964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3982, 'num_write_kb': 15922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 277}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:06.165 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'4151", 'reported_seq': 4292, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:03.438583+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:03.438583+0000', 'last_peered': '2024-04-22T08:28:03.438583+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:03.438583+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 4151, 'log_dups_size': 0, 'ondisk_log_size': 4151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16955812, 'num_objects': 4143, 'num_object_clones': 0, 'num_object_copies': 8286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4148, 'num_write_kb': 16563, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 262}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:06.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:06.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 54 20 67196 67176 4.85834 2.40234 0.0262543 0.0160727 2024-04-22T08:28:07.241 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:07.241 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/130724 objects degraded (31.979%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":122,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":65362,"data_bytes":267677525,"bytes_used":898293760,"bytes_avail":289011998720,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":130724,"degraded_ratio":0.31978825617331169,"misplaced_objects":3632,"misplaced_total":130724,"misplaced_ratio":0.027783727548116643,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":60183,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":606208,"num_keys_recovered":0,"write_bytes_sec":501394,"read_op_per_sec":0,"write_op_per_sec":122},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:07.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:07.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:07.822 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 55 20 67811 67791 4.81368 2.40234 0.0219515 0.0162224 2024-04-22T08:28:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:08.099 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:08.130 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-04-22T08:28:06.152110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267677525,"num_objects":65362,"num_object_clones":0,"num_object_copies":130724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":65362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38964,"num_write_kb":155769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65489,"ondisk_log_size":65489,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":877240,"kb_used_data":517472,"kb_used_omap":12,"kb_used_meta":359731,"kb_avail":282238280,"statfs":{"total":289910292480,"available":289011998720,"internally_reserved":0,"allocated":529891328,"data_stored":529218207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":368364654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,4,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20457669,"num_objects":4992,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":-26369,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1229,"num_write_kb":4932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":606208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.072643"},"pg_stats":[{"pgid":"2.f","version":"35'4074","reported_seq":4134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.378441+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:03.378441+0000","last_peered":"2024-04-22T08:28:03.378441+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:03.378441+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":4074,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":16296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4137","reported_seq":4249,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:04.472986+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:04.472986+0000","last_peered":"2024-04-22T08:28:04.472986+0000","last_clean":"2024-04-22T08:28:04.472986+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:04.472986+0000","last_undegraded":"2024-04-22T08:28:04.472986+0000","last_fullsized":"2024-04-22T08:28:04.472986+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16920576,"num_objects":4131,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":4772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4090","reported_seq":4206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:04.497150+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:04.497150+0000","last_peered":"2024-04-22T08:28:04.497150+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:04.497150+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16736256,"num_objects":4086,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":917,"num_write_kb":3668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4178","reported_seq":4205,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:04.458768+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:04.458768+0000","last_peered":"2024-04-22T08:28:04.458768+0000","last_clean":"2024-04-22T08:28:04.458768+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:04.458768+0000","last_undegraded":"2024-04-22T08:28:04.458768+0000","last_fullsized":"2024-04-22T08:28:04.458768+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4178,"log_dups_size":0,"ondisk_log_size":4178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17113088,"num_objects":4178,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":16712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4069","reported_seq":4204,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.396436+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:03.396436+0000","last_peered":"2024-04-22T08:28:03.396436+0000","last_clean":"2024-04-22T08:28:03.396436+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:03.396436+0000","last_undegraded":"2024-04-22T08:28:03.396436+0000","last_fullsized":"2024-04-22T08:28:03.396436+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16637952,"num_objects":4062,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":4644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4159","reported_seq":4318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.459788+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:03.459788+0000","last_peered":"2024-04-22T08:28:03.459788+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:03.459788+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17014784,"num_objects":4154,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4047","reported_seq":4503,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:03.425604+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:03.425604+0000","last_peered":"2024-04-22T08:28:03.425604+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:03.425604+0000","last_undegraded":"2024-04-22T08:28:03.425604+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":409},{"shards":"0,2","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4021","reported_seq":4162,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:03.433626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:03.433626+0000","last_peered":"2024-04-22T08:28:03.433626+0000","last_clean":"2024-04-22T08:28:03.433626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:03.433626+0000","last_undegraded":"2024-04-22T08:28:03.433626+0000","last_fullsized":"2024-04-22T08:28:03.433626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4252","reported_seq":4372,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:04.406694+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:04.406694+0000","last_peered":"2024-04-22T08:28:04.406694+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:04.406694+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17151186,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":16752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4032","reported_seq":4106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.388967+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:03.388967+0000","last_peered":"2024-04-22T08:28:03.388967+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:03.388967+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16483538,"num_objects":4026,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":16100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":255},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4046","reported_seq":4189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:04.488755+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:04.488755+0000","last_peered":"2024-04-22T08:28:04.488755+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:04.488755+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4134","reported_seq":4186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:04.460374+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:04.460374+0000","last_peered":"2024-04-22T08:28:04.460374+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:04.460374+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16909522,"num_objects":4130,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":16515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4009","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.428369+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:03.428369+0000","last_peered":"2024-04-22T08:28:03.428369+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:03.428369+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413289,"num_objects":4008,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":16029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4106","reported_seq":4276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.425177+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:03.425177+0000","last_peered":"2024-04-22T08:28:03.425177+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:03.425177+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":1348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":261},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'3984","reported_seq":4055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.431849+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:03.431849+0000","last_peered":"2024-04-22T08:28:03.431849+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:03.431849+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303314,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":15922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":277},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4151","reported_seq":4292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:03.438583+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:03.438583+0000","last_peered":"2024-04-22T08:28:03.438583+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:03.438583+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16955812,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":262},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":267677525,"num_objects":65362,"num_object_clones":0,"num_object_copies":130724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3632,"num_objects_unfound":0,"num_objects_dirty":65362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38964,"num_write_kb":155769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":627305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":531668992,"data_stored":531582017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65489,"ondisk_log_size":65489,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392688,"kb_used_data":252776,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":93979152,"statfs":{"total":96636764160,"available":96234651648,"internally_reserved":0,"allocated":258842624,"data_stored":258641609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]}]},{"osd":1,"up_from":33,"seq":141733920771,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85948,"kb_used_data":10036,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94285892,"statfs":{"total":96636764160,"available":96548753408,"internally_reserved":0,"allocated":10276864,"data_stored":9999311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":77721292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313236,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398604,"kb_used_data":254660,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93973236,"statfs":{"total":96636764160,"available":96228593664,"internally_reserved":0,"allocated":260771840,"data_stored":260577287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262242304,"data_stored":262238825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10182656,"data_stored":10102639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259244032,"data_stored":259240553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:08.823 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 56 20 68456 68436 4.77271 2.51953 0.0355682 0.0163633 2024-04-22T08:28:09.824 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 57 20 69114 69094 4.73402 2.57031 0.0169244 0.0164946 2024-04-22T08:28:10.823 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 58 20 69776 69756 4.69698 2.58594 0.0334559 0.0166217 2024-04-22T08:28:11.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:11.826 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 59 20 70406 70386 4.65887 2.46094 0.0475222 0.0167607 2024-04-22T08:28:12.116 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:12.217 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-04-22T08:28:10.154755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280526677,"num_objects":68499,"num_object_clones":0,"num_object_copies":136998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":68499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42101,"num_write_kb":168317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1061481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68626,"ondisk_log_size":68626,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":894736,"kb_used_data":543160,"kb_used_omap":12,"kb_used_meta":351539,"kb_avail":282220784,"statfs":{"total":289910292480,"available":288994082816,"internally_reserved":0,"allocated":556195840,"data_stored":555522719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":359976046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33306821,"num_objects":8129,"num_object_clones":0,"num_object_copies":16258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":-26475,"num_objects_unfound":0,"num_objects_dirty":8129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":17480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004768"},"pg_stats":[{"pgid":"2.f","version":"35'4272","reported_seq":4331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.519237+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:08.519237+0000","last_peered":"2024-04-22T08:28:08.519237+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:08.519237+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":17088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":478},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4328","reported_seq":4440,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.505597+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:09.505597+0000","last_peered":"2024-04-22T08:28:09.505597+0000","last_clean":"2024-04-22T08:28:09.505597+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:09.505597+0000","last_undegraded":"2024-04-22T08:28:09.505597+0000","last_fullsized":"2024-04-22T08:28:09.505597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17702912,"num_objects":4322,"num_object_clones":0,"num_object_copies":8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":5536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4299","reported_seq":4419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.481437+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:09.481437+0000","last_peered":"2024-04-22T08:28:09.481437+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:09.481437+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17592320,"num_objects":4295,"num_object_clones":0,"num_object_copies":8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1126,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4377","reported_seq":4404,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.410008+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:09.410008+0000","last_peered":"2024-04-22T08:28:09.410008+0000","last_clean":"2024-04-22T08:28:09.410008+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:09.410008+0000","last_undegraded":"2024-04-22T08:28:09.410008+0000","last_fullsized":"2024-04-22T08:28:09.410008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17928192,"num_objects":4377,"num_object_clones":0,"num_object_copies":8754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":17508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4270","reported_seq":4405,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:08.473771+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:08.473771+0000","last_peered":"2024-04-22T08:28:08.473771+0000","last_clean":"2024-04-22T08:28:08.473771+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:08.473771+0000","last_undegraded":"2024-04-22T08:28:08.473771+0000","last_fullsized":"2024-04-22T08:28:08.473771+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17461248,"num_objects":4263,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":5448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4355","reported_seq":4513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.513858+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:08.513858+0000","last_peered":"2024-04-22T08:28:08.513858+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:08.513858+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17817600,"num_objects":4350,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":4896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4247","reported_seq":4911,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:08.523434+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:08.523434+0000","last_peered":"2024-04-22T08:28:08.523434+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:08.523434+0000","last_undegraded":"2024-04-22T08:28:08.523434+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17371136,"num_objects":4241,"num_object_clones":0,"num_object_copies":8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":2460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":715},{"shards":"0,2","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4190","reported_seq":4331,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:08.396060+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:08.396060+0000","last_peered":"2024-04-22T08:28:08.396060+0000","last_clean":"2024-04-22T08:28:08.396060+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:08.396060+0000","last_undegraded":"2024-04-22T08:28:08.396060+0000","last_fullsized":"2024-04-22T08:28:08.396060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17154048,"num_objects":4188,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4452","reported_seq":4571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.506774+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:09.506774+0000","last_peered":"2024-04-22T08:28:09.506774+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:09.506774+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17970386,"num_objects":4389,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":17552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4238","reported_seq":4312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.422319+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:08.422319+0000","last_peered":"2024-04-22T08:28:08.422319+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:08.422319+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17327314,"num_objects":4232,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4238","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.463778+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:09.463778+0000","last_peered":"2024-04-22T08:28:09.463778+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:09.463778+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17338368,"num_objects":4233,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4329","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.441426+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:09.441426+0000","last_peered":"2024-04-22T08:28:09.441426+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:09.441426+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17708242,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":17295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4194","reported_seq":4270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.519114+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:08.519114+0000","last_peered":"2024-04-22T08:28:08.519114+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:08.519114+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17171049,"num_objects":4193,"num_object_clones":0,"num_object_copies":8386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":16769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":472},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4286","reported_seq":4458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.497154+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:08.497154+0000","last_peered":"2024-04-22T08:28:08.497154+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:08.497154+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17522688,"num_objects":4278,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":2068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":441},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4185","reported_seq":4258,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.492328+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:08.492328+0000","last_peered":"2024-04-22T08:28:08.492328+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:08.492328+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126610,"num_objects":4183,"num_object_clones":0,"num_object_copies":8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":478},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4366","reported_seq":4507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.502506+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:08.502506+0000","last_peered":"2024-04-22T08:28:08.502506+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:08.502506+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17836452,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4363,"num_write_kb":17423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":477},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":280526677,"num_objects":68499,"num_object_clones":0,"num_object_copies":136998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":68499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42101,"num_write_kb":168317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1061481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558084096,"data_stored":557997121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68626,"ondisk_log_size":68626,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407088,"kb_used_data":258984,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93964752,"statfs":{"total":96636764160,"available":96219906048,"internally_reserved":0,"allocated":265199616,"data_stored":264998601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.505},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920772,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87960,"kb_used_data":20240,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94283880,"statfs":{"total":96636764160,"available":96546693120,"internally_reserved":0,"allocated":20725760,"data_stored":20448207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":69332684},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313237,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":263936,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":270270464,"data_stored":270075911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":138997459},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.373},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.46899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271908864,"data_stored":271905385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20525056,"data_stored":20445039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265650176,"data_stored":265646697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:12.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:12.656 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:12.656 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:12.673 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-04-22T08:28:12.155198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280526677,"num_objects":68499,"num_object_clones":0,"num_object_copies":136998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":68499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42101,"num_write_kb":168317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1061481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68626,"ondisk_log_size":68626,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":935236,"kb_used_data":553324,"kb_used_omap":12,"kb_used_meta":381875,"kb_avail":282180284,"statfs":{"total":289910292480,"available":288952610816,"internally_reserved":0,"allocated":566603776,"data_stored":565930655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":391040110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,5,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26005504,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-218,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":25396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004553"},"pg_stats":[{"pgid":"2.f","version":"35'4272","reported_seq":4331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.519237+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:08.519237+0000","last_peered":"2024-04-22T08:28:08.519237+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:08.519237+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":17088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":478},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4328","reported_seq":4440,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.505597+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:09.505597+0000","last_peered":"2024-04-22T08:28:09.505597+0000","last_clean":"2024-04-22T08:28:09.505597+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:09.505597+0000","last_undegraded":"2024-04-22T08:28:09.505597+0000","last_fullsized":"2024-04-22T08:28:09.505597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17702912,"num_objects":4322,"num_object_clones":0,"num_object_copies":8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":5536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4299","reported_seq":4419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.481437+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:09.481437+0000","last_peered":"2024-04-22T08:28:09.481437+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:09.481437+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17592320,"num_objects":4295,"num_object_clones":0,"num_object_copies":8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1126,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4377","reported_seq":4404,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.410008+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:09.410008+0000","last_peered":"2024-04-22T08:28:09.410008+0000","last_clean":"2024-04-22T08:28:09.410008+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:09.410008+0000","last_undegraded":"2024-04-22T08:28:09.410008+0000","last_fullsized":"2024-04-22T08:28:09.410008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17928192,"num_objects":4377,"num_object_clones":0,"num_object_copies":8754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":17508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4270","reported_seq":4405,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:08.473771+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:08.473771+0000","last_peered":"2024-04-22T08:28:08.473771+0000","last_clean":"2024-04-22T08:28:08.473771+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:08.473771+0000","last_undegraded":"2024-04-22T08:28:08.473771+0000","last_fullsized":"2024-04-22T08:28:08.473771+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17461248,"num_objects":4263,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":5448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4355","reported_seq":4513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.513858+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:08.513858+0000","last_peered":"2024-04-22T08:28:08.513858+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:08.513858+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17817600,"num_objects":4350,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":4896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4247","reported_seq":4911,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:08.523434+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:08.523434+0000","last_peered":"2024-04-22T08:28:08.523434+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:08.523434+0000","last_undegraded":"2024-04-22T08:28:08.523434+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17371136,"num_objects":4241,"num_object_clones":0,"num_object_copies":8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":2460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":715},{"shards":"0,2","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4190","reported_seq":4331,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:08.396060+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:08.396060+0000","last_peered":"2024-04-22T08:28:08.396060+0000","last_clean":"2024-04-22T08:28:08.396060+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:08.396060+0000","last_undegraded":"2024-04-22T08:28:08.396060+0000","last_fullsized":"2024-04-22T08:28:08.396060+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17154048,"num_objects":4188,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4452","reported_seq":4571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.506774+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:09.506774+0000","last_peered":"2024-04-22T08:28:09.506774+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:09.506774+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17970386,"num_objects":4389,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":17552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4238","reported_seq":4312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.422319+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:08.422319+0000","last_peered":"2024-04-22T08:28:08.422319+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:08.422319+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17327314,"num_objects":4232,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4238","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.463778+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:09.463778+0000","last_peered":"2024-04-22T08:28:09.463778+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:09.463778+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17338368,"num_objects":4233,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4329","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.441426+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:09.441426+0000","last_peered":"2024-04-22T08:28:09.441426+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:09.441426+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17708242,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":17295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4194","reported_seq":4270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.519114+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:08.519114+0000","last_peered":"2024-04-22T08:28:08.519114+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:08.519114+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17171049,"num_objects":4193,"num_object_clones":0,"num_object_copies":8386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":16769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":472},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4286","reported_seq":4458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.497154+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:08.497154+0000","last_peered":"2024-04-22T08:28:08.497154+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:08.497154+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17522688,"num_objects":4278,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":2068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":441},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4185","reported_seq":4258,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.492328+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:08.492328+0000","last_peered":"2024-04-22T08:28:08.492328+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:08.492328+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17126610,"num_objects":4183,"num_object_clones":0,"num_object_copies":8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":478},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4366","reported_seq":4507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:08.502506+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:08.502506+0000","last_peered":"2024-04-22T08:28:08.502506+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:08.502506+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17836452,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4363,"num_write_kb":17423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":477},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":280526677,"num_objects":68499,"num_object_clones":0,"num_object_copies":136998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":68499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42101,"num_write_kb":168317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1061481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":568520704,"data_stored":568433729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68626,"ondisk_log_size":68626,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407088,"kb_used_data":258984,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93964752,"statfs":{"total":96636764160,"available":96219906048,"internally_reserved":0,"allocated":265199616,"data_stored":264998601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.505},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920773,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128460,"kb_used_data":30404,"kb_used_omap":4,"kb_used_meta":98043,"kb_avail":94243380,"statfs":{"total":96636764160,"available":96505221120,"internally_reserved":0,"allocated":31133696,"data_stored":30856143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":100396748},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313237,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":263936,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":270270464,"data_stored":270075911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":138997459},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.373},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.46899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271908864,"data_stored":271905385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30961664,"data_stored":30881647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265650176,"data_stored":265646697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:12.674 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:12.674 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'4272", 'reported_seq': 4331, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.519237+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:08.519237+0000', 'last_peered': '2024-04-22T08:28:08.519237+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:08.519237+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4272, 'log_dups_size': 0, 'ondisk_log_size': 4272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17498112, 'num_objects': 4272, 'num_object_clones': 0, 'num_object_copies': 8544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4272, 'num_write_kb': 17088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 478}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.674 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:12.674 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'4299", 'reported_seq': 4419, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:09.481437+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:09.481437+0000', 'last_peered': '2024-04-22T08:28:09.481437+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:09.481437+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 4299, 'log_dups_size': 0, 'ondisk_log_size': 4299, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17592320, 'num_objects': 4295, 'num_object_clones': 0, 'num_object_copies': 8590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1126, 'num_write_kb': 4504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'4355", 'reported_seq': 4513, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.513858+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:08.513858+0000', 'last_peered': '2024-04-22T08:28:08.513858+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:08.513858+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 4355, 'log_dups_size': 0, 'ondisk_log_size': 4355, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17817600, 'num_objects': 4350, 'num_object_clones': 0, 'num_object_copies': 8700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1224, 'num_write_kb': 4896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 471}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'4247", 'reported_seq': 4911, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:08.523434+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:08.523434+0000', 'last_peered': '2024-04-22T08:28:08.523434+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:08.523434+0000', 'last_undegraded': '2024-04-22T08:28:08.523434+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 4247, 'log_dups_size': 0, 'ondisk_log_size': 4247, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17371136, 'num_objects': 4241, 'num_object_clones': 0, 'num_object_copies': 8482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3526, 'num_objects_unfound': 0, 'num_objects_dirty': 4241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 615, 'num_write_kb': 2460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 255, 'num_bytes_recovered': 1044480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 715}, {'shards': '0,2', 'objects': 3526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'4452", 'reported_seq': 4571, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:09.506774+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:09.506774+0000', 'last_peered': '2024-04-22T08:28:09.506774+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:09.506774+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 4452, 'log_dups_size': 0, 'ondisk_log_size': 4452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17970386, 'num_objects': 4389, 'num_object_clones': 0, 'num_object_copies': 8778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4392, 'num_write_kb': 17552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'4238", 'reported_seq': 4312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.422319+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:08.422319+0000', 'last_peered': '2024-04-22T08:28:08.422319+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:08.422319+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4238, 'log_dups_size': 0, 'ondisk_log_size': 4238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17327314, 'num_objects': 4232, 'num_object_clones': 0, 'num_object_copies': 8464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4236, 'num_write_kb': 16924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 461}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:12.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'4238", 'reported_seq': 4381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:09.463778+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:09.463778+0000', 'last_peered': '2024-04-22T08:28:09.463778+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:09.463778+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 4238, 'log_dups_size': 0, 'ondisk_log_size': 4238, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17338368, 'num_objects': 4233, 'num_object_clones': 0, 'num_object_copies': 8466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 692, 'num_write_kb': 2768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'4329", 'reported_seq': 4381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:09.441426+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:09.441426+0000', 'last_peered': '2024-04-22T08:28:09.441426+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:09.441426+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4329, 'log_dups_size': 0, 'ondisk_log_size': 4329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17708242, 'num_objects': 4325, 'num_object_clones': 0, 'num_object_copies': 8650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4327, 'num_write_kb': 17295, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 522}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'4194", 'reported_seq': 4270, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.519114+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:08.519114+0000', 'last_peered': '2024-04-22T08:28:08.519114+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:08.519114+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4194, 'log_dups_size': 0, 'ondisk_log_size': 4194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17171049, 'num_objects': 4193, 'num_object_clones': 0, 'num_object_copies': 8386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4193, 'num_write_kb': 16769, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 472}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'4286", 'reported_seq': 4458, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.497154+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:08.497154+0000', 'last_peered': '2024-04-22T08:28:08.497154+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:08.497154+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 4286, 'log_dups_size': 0, 'ondisk_log_size': 4286, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17522688, 'num_objects': 4278, 'num_object_clones': 0, 'num_object_copies': 8556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 517, 'num_write_kb': 2068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 441}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'4185", 'reported_seq': 4258, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.492328+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:08.492328+0000', 'last_peered': '2024-04-22T08:28:08.492328+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:08.492328+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4185, 'log_dups_size': 0, 'ondisk_log_size': 4185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17126610, 'num_objects': 4183, 'num_object_clones': 0, 'num_object_copies': 8366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4183, 'num_write_kb': 16726, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 478}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:12.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'4366", 'reported_seq': 4507, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:08.502506+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:08.502506+0000', 'last_peered': '2024-04-22T08:28:08.502506+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:08.502506+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 4366, 'log_dups_size': 0, 'ondisk_log_size': 4366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17836452, 'num_objects': 4358, 'num_object_clones': 0, 'num_object_copies': 8716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4363, 'num_write_kb': 17423, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 477}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:12.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:12.826 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:28:12.829759+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0169226 2024-04-22T08:28:12.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:28:12.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 60 20 70906 70886 4.61378 1.95312 0.00703127 0.0169226 2024-04-22T08:28:13.529 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:13.529 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/136998 objects degraded (30.514%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":68499,"data_bytes":280526677,"bytes_used":957681664,"bytes_avail":288952610816,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":136998,"degraded_ratio":0.30514314077577775,"misplaced_objects":3526,"misplaced_total":136998,"misplaced_ratio":0.025737602008788451,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":74382,"recovering_keys_per_sec":0,"num_objects_recovered":218,"num_bytes_recovered":892928,"num_keys_recovered":0,"write_bytes_sec":2166303,"read_op_per_sec":0,"write_op_per_sec":528},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:13.581 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:13.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:13.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 61 20 71518 71498 4.5773 2.39062 0.00659061 0.0170593 2024-04-22T08:28:14.526 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:14.526 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:14.605 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-04-22T08:28:14.155555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":288276309,"num_objects":70391,"num_object_clones":0,"num_object_copies":140782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":70391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43993,"num_write_kb":175885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70518,"ondisk_log_size":70518,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":952956,"kb_used_data":562852,"kb_used_omap":12,"kb_used_meta":390067,"kb_avail":282162564,"statfs":{"total":289910292480,"available":288934465536,"internally_reserved":0,"allocated":576360448,"data_stored":575687327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":399428718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,4,7,8,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33755136,"num_objects":8241,"num_object_clones":0,"num_object_copies":16482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-326,"num_objects_unfound":0,"num_objects_dirty":8241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":32964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004435"},"pg_stats":[{"pgid":"2.f","version":"35'4447","reported_seq":4507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.481409+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:13.481409+0000","last_peered":"2024-04-22T08:28:13.481409+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:13.481409+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":653},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4328","reported_seq":4440,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.505597+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:09.505597+0000","last_peered":"2024-04-22T08:28:09.505597+0000","last_clean":"2024-04-22T08:28:09.505597+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:09.505597+0000","last_undegraded":"2024-04-22T08:28:09.505597+0000","last_fullsized":"2024-04-22T08:28:09.505597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17702912,"num_objects":4322,"num_object_clones":0,"num_object_copies":8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":5536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4299","reported_seq":4419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.481437+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:09.481437+0000","last_peered":"2024-04-22T08:28:09.481437+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:09.481437+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17592320,"num_objects":4295,"num_object_clones":0,"num_object_copies":8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1126,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4377","reported_seq":4404,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:09.410008+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:09.410008+0000","last_peered":"2024-04-22T08:28:09.410008+0000","last_clean":"2024-04-22T08:28:09.410008+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:09.410008+0000","last_undegraded":"2024-04-22T08:28:09.410008+0000","last_fullsized":"2024-04-22T08:28:09.410008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17928192,"num_objects":4377,"num_object_clones":0,"num_object_copies":8754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":17508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4454","reported_seq":4589,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.463063+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:13.463063+0000","last_peered":"2024-04-22T08:28:13.463063+0000","last_clean":"2024-04-22T08:28:13.463063+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:13.463063+0000","last_undegraded":"2024-04-22T08:28:13.463063+0000","last_fullsized":"2024-04-22T08:28:13.463063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":6184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4561","reported_seq":4720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.511364+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:13.511364+0000","last_peered":"2024-04-22T08:28:13.511364+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:13.511364+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1430,"num_write_kb":5720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":677},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4432","reported_seq":5317,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:13.523661+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:13.523661+0000","last_peered":"2024-04-22T08:28:13.523661+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:13.523661+0000","last_undegraded":"2024-04-22T08:28:13.523661+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18128896,"num_objects":4426,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":1486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008},{"shards":"0,2","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4395","reported_seq":4536,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.515645+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:13.515645+0000","last_peered":"2024-04-22T08:28:13.515645+0000","last_clean":"2024-04-22T08:28:13.515645+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:13.515645+0000","last_undegraded":"2024-04-22T08:28:13.515645+0000","last_fullsized":"2024-04-22T08:28:13.515645+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17993728,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4452","reported_seq":4571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.506774+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:09.506774+0000","last_peered":"2024-04-22T08:28:09.506774+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:09.506774+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17970386,"num_objects":4389,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":17552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4409","reported_seq":4482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.528263+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:13.528263+0000","last_peered":"2024-04-22T08:28:13.528263+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:13.528263+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18027730,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":632},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4238","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.463778+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:09.463778+0000","last_peered":"2024-04-22T08:28:09.463778+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:09.463778+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17338368,"num_objects":4233,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4329","reported_seq":4381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:09.441426+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:09.441426+0000","last_peered":"2024-04-22T08:28:09.441426+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:09.441426+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17708242,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":17295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4382","reported_seq":4457,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.525876+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:13.525876+0000","last_peered":"2024-04-22T08:28:13.525876+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:13.525876+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17941097,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":17521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":660},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4470","reported_seq":4642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.533518+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:13.533518+0000","last_peered":"2024-04-22T08:28:13.533518+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:13.533518+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18276352,"num_objects":4462,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":2804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":625},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4410","reported_seq":4481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.536892+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:13.536892+0000","last_peered":"2024-04-22T08:28:13.536892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:13.536892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18048210,"num_objects":4408,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":17626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":703},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4535","reported_seq":4676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.452220+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:13.452220+0000","last_peered":"2024-04-22T08:28:13.452220+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:13.452220+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18528676,"num_objects":4527,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":18099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":646},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":288276309,"num_objects":70391,"num_object_clones":0,"num_object_copies":140782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":70391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43993,"num_write_kb":175885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577933312,"data_stored":577846337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70518,"ondisk_log_size":70518,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407088,"kb_used_data":258984,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93964752,"statfs":{"total":96636764160,"available":96219906048,"internally_reserved":0,"allocated":265199616,"data_stored":264998601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.505},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920773,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128460,"kb_used_data":30404,"kb_used_omap":4,"kb_used_meta":98043,"kb_avail":94243380,"statfs":{"total":96636764160,"available":96505221120,"internally_reserved":0,"allocated":31133696,"data_stored":30856143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":100396748},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313238,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417408,"kb_used_data":273464,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93954432,"statfs":{"total":96636764160,"available":96209338368,"internally_reserved":0,"allocated":280027136,"data_stored":279832583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.22},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281317993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30961664,"data_stored":30881647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265650176,"data_stored":265646697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:14.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 62 20 72136 72116 4.54241 2.41406 0.0120452 0.0171918 2024-04-22T08:28:15.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 63 20 72772 72752 4.50975 2.48438 0.0168814 0.0173156 2024-04-22T08:28:16.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 64 20 73406 73386 4.47798 2.47656 0.0304334 0.0174388 2024-04-22T08:28:17.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:17.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 65 20 74006 73986 4.44515 2.34375 0.0248041 0.0175659 2024-04-22T08:28:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:18.397 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:18.437 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-22T08:28:18.157162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293011285,"num_objects":71547,"num_object_clones":0,"num_object_copies":143094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":71547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45149,"num_write_kb":180509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71674,"ondisk_log_size":71674,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":960232,"kb_used_data":576656,"kb_used_omap":12,"kb_used_meta":383539,"kb_avail":282155288,"statfs":{"total":289910292480,"available":288927014912,"internally_reserved":0,"allocated":590495744,"data_stored":589822623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":392744046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,6,7,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25333760,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":24740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005052"},"pg_stats":[{"pgid":"2.f","version":"35'4447","reported_seq":4507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.481409+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:13.481409+0000","last_peered":"2024-04-22T08:28:13.481409+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:13.481409+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":653},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4530","reported_seq":4642,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:14.426562+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:14.426562+0000","last_peered":"2024-04-22T08:28:14.426562+0000","last_clean":"2024-04-22T08:28:14.426562+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:14.426562+0000","last_undegraded":"2024-04-22T08:28:14.426562+0000","last_fullsized":"2024-04-22T08:28:14.426562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":4524,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1586,"num_write_kb":6344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4485","reported_seq":4604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.480779+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:14.480779+0000","last_peered":"2024-04-22T08:28:14.480779+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:14.480779+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18354176,"num_objects":4481,"num_object_clones":0,"num_object_copies":8962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1312,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":726}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4578","reported_seq":4605,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:14.461741+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:14.461741+0000","last_peered":"2024-04-22T08:28:14.461741+0000","last_clean":"2024-04-22T08:28:14.461741+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:14.461741+0000","last_undegraded":"2024-04-22T08:28:14.461741+0000","last_fullsized":"2024-04-22T08:28:14.461741+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":4578,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4578,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4454","reported_seq":4589,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.463063+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:13.463063+0000","last_peered":"2024-04-22T08:28:13.463063+0000","last_clean":"2024-04-22T08:28:13.463063+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:13.463063+0000","last_undegraded":"2024-04-22T08:28:13.463063+0000","last_fullsized":"2024-04-22T08:28:13.463063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":6184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4561","reported_seq":4720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.511364+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:13.511364+0000","last_peered":"2024-04-22T08:28:13.511364+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:13.511364+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1430,"num_write_kb":5720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":677},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4432","reported_seq":5317,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:13.523661+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:13.523661+0000","last_peered":"2024-04-22T08:28:13.523661+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:13.523661+0000","last_undegraded":"2024-04-22T08:28:13.523661+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18128896,"num_objects":4426,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":1486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008},{"shards":"0,2","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4395","reported_seq":4536,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.515645+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:13.515645+0000","last_peered":"2024-04-22T08:28:13.515645+0000","last_clean":"2024-04-22T08:28:13.515645+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:13.515645+0000","last_undegraded":"2024-04-22T08:28:13.515645+0000","last_fullsized":"2024-04-22T08:28:13.515645+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17993728,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4641","reported_seq":4761,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.469182+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:14.469182+0000","last_peered":"2024-04-22T08:28:14.469182+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:14.469182+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18744530,"num_objects":4578,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":18308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4409","reported_seq":4482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.528263+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:13.528263+0000","last_peered":"2024-04-22T08:28:13.528263+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:13.528263+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18027730,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":632},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4432","reported_seq":4575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.484152+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:14.484152+0000","last_peered":"2024-04-22T08:28:14.484152+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:14.484152+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18132992,"num_objects":4427,"num_object_clones":0,"num_object_copies":8854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":3544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":677}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4513","reported_seq":4565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.503975+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:14.503975+0000","last_peered":"2024-04-22T08:28:14.503975+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:14.503975+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461906,"num_objects":4509,"num_object_clones":0,"num_object_copies":9018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":706}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4382","reported_seq":4457,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.525876+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:13.525876+0000","last_peered":"2024-04-22T08:28:13.525876+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:13.525876+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17941097,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":17521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":660},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4470","reported_seq":4642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.533518+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:13.533518+0000","last_peered":"2024-04-22T08:28:13.533518+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:13.533518+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18276352,"num_objects":4462,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":2804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":625},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4410","reported_seq":4481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.536892+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:13.536892+0000","last_peered":"2024-04-22T08:28:13.536892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:13.536892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18048210,"num_objects":4408,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":17626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":703},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4535","reported_seq":4676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.452220+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:13.452220+0000","last_peered":"2024-04-22T08:28:13.452220+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:13.452220+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18528676,"num_objects":4527,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":18099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":646},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293011285,"num_objects":71547,"num_object_clones":0,"num_object_copies":143094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":71547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45149,"num_write_kb":180509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593940480,"data_stored":593853505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71674,"ondisk_log_size":71674,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400956,"kb_used_data":265140,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93970884,"statfs":{"total":96636764160,"available":96226185216,"internally_reserved":0,"allocated":271503360,"data_stored":271302345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.877}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5289999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920774,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141868,"kb_used_data":38052,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94229972,"statfs":{"total":96636764160,"available":96491491328,"internally_reserved":0,"allocated":38965248,"data_stored":38687695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":106294988},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313238,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417408,"kb_used_data":273464,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93954432,"statfs":{"total":96636764160,"available":96209338368,"internally_reserved":0,"allocated":280027136,"data_stored":279832583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.22},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281317993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40730624,"data_stored":40650607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271888384,"data_stored":271884905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:18.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:18.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 66 20 74585 74565 4.41207 2.26172 0.0358971 0.0176993 2024-04-22T08:28:19.212 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:19.212 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-22T08:28:18.157162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293011285,"num_objects":71547,"num_object_clones":0,"num_object_copies":143094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":71547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45149,"num_write_kb":180509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71674,"ondisk_log_size":71674,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":960232,"kb_used_data":576656,"kb_used_omap":12,"kb_used_meta":383539,"kb_avail":282155288,"statfs":{"total":289910292480,"available":288927014912,"internally_reserved":0,"allocated":590495744,"data_stored":589822623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":392744046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,6,7,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25333760,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":24740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005052"},"pg_stats":[{"pgid":"2.f","version":"35'4447","reported_seq":4507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.481409+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:13.481409+0000","last_peered":"2024-04-22T08:28:13.481409+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:13.481409+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":653},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4530","reported_seq":4642,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:14.426562+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:14.426562+0000","last_peered":"2024-04-22T08:28:14.426562+0000","last_clean":"2024-04-22T08:28:14.426562+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:14.426562+0000","last_undegraded":"2024-04-22T08:28:14.426562+0000","last_fullsized":"2024-04-22T08:28:14.426562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":4524,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1586,"num_write_kb":6344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4485","reported_seq":4604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.480779+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:14.480779+0000","last_peered":"2024-04-22T08:28:14.480779+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:14.480779+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18354176,"num_objects":4481,"num_object_clones":0,"num_object_copies":8962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1312,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":726}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4578","reported_seq":4605,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:14.461741+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:14.461741+0000","last_peered":"2024-04-22T08:28:14.461741+0000","last_clean":"2024-04-22T08:28:14.461741+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:14.461741+0000","last_undegraded":"2024-04-22T08:28:14.461741+0000","last_fullsized":"2024-04-22T08:28:14.461741+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":4578,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4578,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4454","reported_seq":4589,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.463063+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:13.463063+0000","last_peered":"2024-04-22T08:28:13.463063+0000","last_clean":"2024-04-22T08:28:13.463063+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:13.463063+0000","last_undegraded":"2024-04-22T08:28:13.463063+0000","last_fullsized":"2024-04-22T08:28:13.463063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18214912,"num_objects":4447,"num_object_clones":0,"num_object_copies":8894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":6184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4561","reported_seq":4720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.511364+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:13.511364+0000","last_peered":"2024-04-22T08:28:13.511364+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:13.511364+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1430,"num_write_kb":5720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":677},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4432","reported_seq":5317,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:13.523661+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:13.523661+0000","last_peered":"2024-04-22T08:28:13.523661+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:13.523661+0000","last_undegraded":"2024-04-22T08:28:13.523661+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18128896,"num_objects":4426,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":1486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008},{"shards":"0,2","objects":3418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4395","reported_seq":4536,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:13.515645+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:13.515645+0000","last_peered":"2024-04-22T08:28:13.515645+0000","last_clean":"2024-04-22T08:28:13.515645+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:13.515645+0000","last_undegraded":"2024-04-22T08:28:13.515645+0000","last_fullsized":"2024-04-22T08:28:13.515645+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17993728,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4641","reported_seq":4761,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.469182+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:14.469182+0000","last_peered":"2024-04-22T08:28:14.469182+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:14.469182+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18744530,"num_objects":4578,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":18308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4409","reported_seq":4482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.528263+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:13.528263+0000","last_peered":"2024-04-22T08:28:13.528263+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:13.528263+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18027730,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":632},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4432","reported_seq":4575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.484152+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:14.484152+0000","last_peered":"2024-04-22T08:28:14.484152+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:14.484152+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18132992,"num_objects":4427,"num_object_clones":0,"num_object_copies":8854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":3544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":677}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4513","reported_seq":4565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:14.503975+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:14.503975+0000","last_peered":"2024-04-22T08:28:14.503975+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:14.503975+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461906,"num_objects":4509,"num_object_clones":0,"num_object_copies":9018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":706}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4382","reported_seq":4457,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.525876+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:13.525876+0000","last_peered":"2024-04-22T08:28:13.525876+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:13.525876+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17941097,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":17521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":660},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4470","reported_seq":4642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.533518+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:13.533518+0000","last_peered":"2024-04-22T08:28:13.533518+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:13.533518+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18276352,"num_objects":4462,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":2804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":625},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4410","reported_seq":4481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.536892+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:13.536892+0000","last_peered":"2024-04-22T08:28:13.536892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:13.536892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18048210,"num_objects":4408,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":17626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":703},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4535","reported_seq":4676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:13.452220+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:13.452220+0000","last_peered":"2024-04-22T08:28:13.452220+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:13.452220+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18528676,"num_objects":4527,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":18099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":646},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293011285,"num_objects":71547,"num_object_clones":0,"num_object_copies":143094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3418,"num_objects_unfound":0,"num_objects_dirty":71547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45149,"num_write_kb":180509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":1503849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":593940480,"data_stored":593853505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71674,"ondisk_log_size":71674,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400956,"kb_used_data":265140,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93970884,"statfs":{"total":96636764160,"available":96226185216,"internally_reserved":0,"allocated":271503360,"data_stored":271302345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.877}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5289999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920774,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141868,"kb_used_data":38052,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94229972,"statfs":{"total":96636764160,"available":96491491328,"internally_reserved":0,"allocated":38965248,"data_stored":38687695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":106294988},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313238,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417408,"kb_used_data":273464,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93954432,"statfs":{"total":96636764160,"available":96209338368,"internally_reserved":0,"allocated":280027136,"data_stored":279832583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":147386067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.22},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281321472,"data_stored":281317993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40730624,"data_stored":40650607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271888384,"data_stored":271884905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'4447", 'reported_seq': 4507, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.481409+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:13.481409+0000', 'last_peered': '2024-04-22T08:28:13.481409+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:13.481409+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4447, 'log_dups_size': 0, 'ondisk_log_size': 4447, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18214912, 'num_objects': 4447, 'num_object_clones': 0, 'num_object_copies': 8894, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4447, 'num_write_kb': 17788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 653}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'4485", 'reported_seq': 4604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:14.480779+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:14.480779+0000', 'last_peered': '2024-04-22T08:28:14.480779+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:14.480779+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 4485, 'log_dups_size': 0, 'ondisk_log_size': 4485, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18354176, 'num_objects': 4481, 'num_object_clones': 0, 'num_object_copies': 8962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1312, 'num_write_kb': 5248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:19.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'4561", 'reported_seq': 4720, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.511364+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:13.511364+0000', 'last_peered': '2024-04-22T08:28:13.511364+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:13.511364+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 4561, 'log_dups_size': 0, 'ondisk_log_size': 4561, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18661376, 'num_objects': 4556, 'num_object_clones': 0, 'num_object_copies': 9112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1430, 'num_write_kb': 5720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 677}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'4432", 'reported_seq': 5317, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:13.523661+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:13.523661+0000', 'last_peered': '2024-04-22T08:28:13.523661+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:13.523661+0000', 'last_undegraded': '2024-04-22T08:28:13.523661+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 4432, 'log_dups_size': 0, 'ondisk_log_size': 4432, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18128896, 'num_objects': 4426, 'num_object_clones': 0, 'num_object_copies': 8852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3418, 'num_objects_unfound': 0, 'num_objects_dirty': 4426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 800, 'num_write_kb': 3200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 363, 'num_bytes_recovered': 1486848, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1008}, {'shards': '0,2', 'objects': 3418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'4641", 'reported_seq': 4761, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:14.469182+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:14.469182+0000', 'last_peered': '2024-04-22T08:28:14.469182+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:14.469182+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 4641, 'log_dups_size': 0, 'ondisk_log_size': 4641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18744530, 'num_objects': 4578, 'num_object_clones': 0, 'num_object_copies': 9156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4581, 'num_write_kb': 18308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'4409", 'reported_seq': 4482, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.528263+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:13.528263+0000', 'last_peered': '2024-04-22T08:28:13.528263+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:13.528263+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4409, 'log_dups_size': 0, 'ondisk_log_size': 4409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18027730, 'num_objects': 4403, 'num_object_clones': 0, 'num_object_copies': 8806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4407, 'num_write_kb': 17608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 632}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'4432", 'reported_seq': 4575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:14.484152+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:14.484152+0000', 'last_peered': '2024-04-22T08:28:14.484152+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:14.484152+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 4432, 'log_dups_size': 0, 'ondisk_log_size': 4432, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18132992, 'num_objects': 4427, 'num_object_clones': 0, 'num_object_copies': 8854, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 886, 'num_write_kb': 3544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 677}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:19.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'4513", 'reported_seq': 4565, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:14.503975+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:14.503975+0000', 'last_peered': '2024-04-22T08:28:14.503975+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:14.503975+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4513, 'log_dups_size': 0, 'ondisk_log_size': 4513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18461906, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 9018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4511, 'num_write_kb': 18031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 706}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'4382", 'reported_seq': 4457, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.525876+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:13.525876+0000', 'last_peered': '2024-04-22T08:28:13.525876+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:13.525876+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4382, 'log_dups_size': 0, 'ondisk_log_size': 4382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17941097, 'num_objects': 4381, 'num_object_clones': 0, 'num_object_copies': 8762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4381, 'num_write_kb': 17521, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 660}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'4470", 'reported_seq': 4642, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.533518+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:13.533518+0000', 'last_peered': '2024-04-22T08:28:13.533518+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:13.533518+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 4470, 'log_dups_size': 0, 'ondisk_log_size': 4470, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18276352, 'num_objects': 4462, 'num_object_clones': 0, 'num_object_copies': 8924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 701, 'num_write_kb': 2804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 625}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'4410", 'reported_seq': 4481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.536892+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:13.536892+0000', 'last_peered': '2024-04-22T08:28:13.536892+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:13.536892+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4410, 'log_dups_size': 0, 'ondisk_log_size': 4410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18048210, 'num_objects': 4408, 'num_object_clones': 0, 'num_object_copies': 8816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4408, 'num_write_kb': 17626, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 703}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'4535", 'reported_seq': 4676, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:13.452220+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:13.452220+0000', 'last_peered': '2024-04-22T08:28:13.452220+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:13.452220+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 4535, 'log_dups_size': 0, 'ondisk_log_size': 4535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18528676, 'num_objects': 4527, 'num_object_clones': 0, 'num_object_copies': 9054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4532, 'num_write_kb': 18099, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 646}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:19.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:19.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 67 20 75171 75151 4.38039 2.28906 0.0789556 0.0178267 2024-04-22T08:28:20.288 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:20.288 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/149276 objects degraded (28.005%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":135,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":74638,"data_bytes":305672021,"bytes_used":1000247296,"bytes_avail":288910045184,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":149276,"degraded_ratio":0.28004501728342129,"misplaced_objects":3312,"misplaced_total":149276,"misplaced_ratio":0.022187089686218811,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":109195,"recovering_keys_per_sec":0,"num_objects_recovered":320,"num_bytes_recovered":1310720,"num_keys_recovered":0,"write_bytes_sec":3165311,"read_op_per_sec":0,"write_op_per_sec":772},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:20.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:20.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:20.827 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 68 20 75720 75700 4.34751 2.14453 0.035134 0.0179616 2024-04-22T08:28:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:20.950 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:21.006 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-04-22T08:28:20.157588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305672021,"num_objects":74638,"num_object_clones":0,"num_object_copies":149276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3312,"num_objects_unfound":0,"num_objects_dirty":74638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48240,"num_write_kb":192873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":489,"num_bytes_recovered":1938025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74765,"ondisk_log_size":74765,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":976804,"kb_used_data":593228,"kb_used_omap":12,"kb_used_meta":383539,"kb_avail":282138716,"statfs":{"total":289910292480,"available":288910045184,"internally_reserved":0,"allocated":607465472,"data_stored":606792351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":392744046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,7,12,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37994496,"num_objects":9276,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-320,"num_objects_unfound":0,"num_objects_dirty":9276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":37104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003398"},"pg_stats":[{"pgid":"2.f","version":"35'4656","reported_seq":4716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.484067+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:18.484067+0000","last_peered":"2024-04-22T08:28:18.484067+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:18.484067+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":862},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4699","reported_seq":4811,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.503886+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:19.503886+0000","last_peered":"2024-04-22T08:28:19.503886+0000","last_clean":"2024-04-22T08:28:19.503886+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:19.503886+0000","last_undegraded":"2024-04-22T08:28:19.503886+0000","last_fullsized":"2024-04-22T08:28:19.503886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19222528,"num_objects":4693,"num_object_clones":0,"num_object_copies":9386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1755,"num_write_kb":7020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4668","reported_seq":4787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.452720+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:19.452720+0000","last_peered":"2024-04-22T08:28:19.452720+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:19.452720+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":4664,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":5980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4775","reported_seq":4802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.440560+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:19.440560+0000","last_peered":"2024-04-22T08:28:19.440560+0000","last_clean":"2024-04-22T08:28:19.440560+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:19.440560+0000","last_undegraded":"2024-04-22T08:28:19.440560+0000","last_fullsized":"2024-04-22T08:28:19.440560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19558400,"num_objects":4775,"num_object_clones":0,"num_object_copies":9550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4636","reported_seq":4771,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:18.497180+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:18.497180+0000","last_peered":"2024-04-22T08:28:18.497180+0000","last_clean":"2024-04-22T08:28:18.497180+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:18.497180+0000","last_undegraded":"2024-04-22T08:28:18.497180+0000","last_fullsized":"2024-04-22T08:28:18.497180+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18960384,"num_objects":4629,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4733","reported_seq":4892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.459220+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:18.459220+0000","last_peered":"2024-04-22T08:28:18.459220+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:18.459220+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19365888,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1602,"num_write_kb":6408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":849},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4627","reported_seq":5724,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:18.543801+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:18.543801+0000","last_peered":"2024-04-22T08:28:18.543801+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:18.543801+0000","last_undegraded":"2024-04-22T08:28:18.543801+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18927616,"num_objects":4621,"num_object_clones":0,"num_object_copies":9242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3312,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":995,"num_write_kb":3980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":1921024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1309},{"shards":"0,2","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4570","reported_seq":4711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:18.484617+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:18.484617+0000","last_peered":"2024-04-22T08:28:18.484617+0000","last_clean":"2024-04-22T08:28:18.484617+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:18.484617+0000","last_undegraded":"2024-04-22T08:28:18.484617+0000","last_fullsized":"2024-04-22T08:28:18.484617+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18710528,"num_objects":4568,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":4872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4851","reported_seq":4971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.458418+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:19.458418+0000","last_peered":"2024-04-22T08:28:19.458418+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:19.458418+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19604690,"num_objects":4788,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4619","reported_seq":4693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.540862+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:18.540862+0000","last_peered":"2024-04-22T08:28:18.540862+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:18.540862+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18887890,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4630","reported_seq":4772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.485673+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:19.485673+0000","last_peered":"2024-04-22T08:28:19.485673+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:19.485673+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18944000,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4705","reported_seq":4757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.470508+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:19.470508+0000","last_peered":"2024-04-22T08:28:19.470508+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:19.470508+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248338,"num_objects":4701,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4703,"num_write_kb":18799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4568","reported_seq":4644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.520715+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:18.520715+0000","last_peered":"2024-04-22T08:28:18.520715+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:18.520715+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18702953,"num_objects":4567,"num_object_clones":0,"num_object_copies":9134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4567,"num_write_kb":18265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":846},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4689","reported_seq":4859,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.572628+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:18.572628+0000","last_peered":"2024-04-22T08:28:18.572628+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:18.572628+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173376,"num_objects":4681,"num_object_clones":0,"num_object_copies":9362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":3680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":844},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4600","reported_seq":4673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.542919+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:18.542919+0000","last_peered":"2024-04-22T08:28:18.542919+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:18.542919+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18826450,"num_objects":4598,"num_object_clones":0,"num_object_copies":9196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4598,"num_write_kb":18386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":893},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4739","reported_seq":4880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:18.509633+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:18.509633+0000","last_peered":"2024-04-22T08:28:18.509633+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:18.509633+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19364260,"num_objects":4731,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":18915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":850},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305672021,"num_objects":74638,"num_object_clones":0,"num_object_copies":149276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3312,"num_objects_unfound":0,"num_objects_dirty":74638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48240,"num_write_kb":192873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":489,"num_bytes_recovered":1938025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":609554432,"data_stored":609467457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74765,"ondisk_log_size":74765,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415020,"kb_used_data":271012,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":93956820,"statfs":{"total":96636764160,"available":96211783680,"internally_reserved":0,"allocated":277516288,"data_stored":277315273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147451599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920774,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141868,"kb_used_data":38052,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94229972,"statfs":{"total":96636764160,"available":96491491328,"internally_reserved":0,"allocated":38965248,"data_stored":38687695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":106294988},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313239,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419916,"kb_used_data":284164,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93951924,"statfs":{"total":96636764160,"available":96206770176,"internally_reserved":0,"allocated":290983936,"data_stored":290789383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":138997459},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.41599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290746368,"data_stored":290742889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40730624,"data_stored":40650607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278077440,"data_stored":278073961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:21.828 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 69 20 76329 76309 4.31898 2.37891 0.01468 0.0180748 2024-04-22T08:28:22.828 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 70 20 76941 76921 4.29143 2.39062 0.0145902 0.0181972 2024-04-22T08:28:23.828 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 71 20 77575 77555 4.26587 2.47656 0.00880364 0.0183038 2024-04-22T08:28:24.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:24.828 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 72 20 78174 78154 4.23912 2.33984 0.0151332 0.0184213 2024-04-22T08:28:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:24.905 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:24.956 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-04-22T08:28:24.161542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313458517,"num_objects":76539,"num_object_clones":0,"num_object_copies":153078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":76539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50141,"num_write_kb":200477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76666,"ondisk_log_size":76666,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1029720,"kb_used_data":611712,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282085800,"statfs":{"total":289910292480,"available":288855859200,"internally_reserved":0,"allocated":626393088,"data_stored":625719967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":428002414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,11,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32931840,"num_objects":8040,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-325,"num_objects_unfound":0,"num_objects_dirty":8040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":32160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":1331200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006344"},"pg_stats":[{"pgid":"2.f","version":"35'4847","reported_seq":4907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.581813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:23.581813+0000","last_peered":"2024-04-22T08:28:23.581813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:23.581813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4847,"num_write_kb":19388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1053},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4699","reported_seq":4811,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.503886+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:19.503886+0000","last_peered":"2024-04-22T08:28:19.503886+0000","last_clean":"2024-04-22T08:28:19.503886+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:19.503886+0000","last_undegraded":"2024-04-22T08:28:19.503886+0000","last_fullsized":"2024-04-22T08:28:19.503886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19222528,"num_objects":4693,"num_object_clones":0,"num_object_copies":9386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1755,"num_write_kb":7020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4668","reported_seq":4787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.452720+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:19.452720+0000","last_peered":"2024-04-22T08:28:19.452720+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:19.452720+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":4664,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":5980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4775","reported_seq":4802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.440560+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:19.440560+0000","last_peered":"2024-04-22T08:28:19.440560+0000","last_clean":"2024-04-22T08:28:19.440560+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:19.440560+0000","last_undegraded":"2024-04-22T08:28:19.440560+0000","last_fullsized":"2024-04-22T08:28:19.440560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19558400,"num_objects":4775,"num_object_clones":0,"num_object_copies":9550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4849","reported_seq":4981,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.566754+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:23.566754+0000","last_peered":"2024-04-22T08:28:23.566754+0000","last_clean":"2024-04-22T08:28:23.566754+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:23.566754+0000","last_undegraded":"2024-04-22T08:28:23.566754+0000","last_fullsized":"2024-04-22T08:28:23.566754+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":7764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4912","reported_seq":5071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.533436+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:23.533436+0000","last_peered":"2024-04-22T08:28:23.533436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:23.533436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20099072,"num_objects":4907,"num_object_clones":0,"num_object_copies":9814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1781,"num_write_kb":7124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1028},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4819","reported_seq":6138,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:23.600538+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:23.600538+0000","last_peered":"2024-04-22T08:28:23.600538+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:23.600538+0000","last_undegraded":"2024-04-22T08:28:23.600538+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19714048,"num_objects":4813,"num_object_clones":0,"num_object_copies":9626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1187,"num_write_kb":4748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1612},{"shards":"0,2","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4772","reported_seq":4913,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.585483+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:23.585483+0000","last_peered":"2024-04-22T08:28:23.585483+0000","last_clean":"2024-04-22T08:28:23.585483+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:23.585483+0000","last_undegraded":"2024-04-22T08:28:23.585483+0000","last_fullsized":"2024-04-22T08:28:23.585483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":4770,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":5680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4851","reported_seq":4971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.458418+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:19.458418+0000","last_peered":"2024-04-22T08:28:19.458418+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:19.458418+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19604690,"num_objects":4788,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4814","reported_seq":4888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.562495+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:23.562495+0000","last_peered":"2024-04-22T08:28:23.562495+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:23.562495+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19686610,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":19228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1037},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4630","reported_seq":4772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.485673+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:19.485673+0000","last_peered":"2024-04-22T08:28:19.485673+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:19.485673+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18944000,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4705","reported_seq":4757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.470508+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:19.470508+0000","last_peered":"2024-04-22T08:28:19.470508+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:19.470508+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248338,"num_objects":4701,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4703,"num_write_kb":18799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4753","reported_seq":4829,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.580364+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:23.580364+0000","last_peered":"2024-04-22T08:28:23.580364+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:23.580364+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19460713,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4752,"num_write_kb":19005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1031},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4866","reported_seq":5037,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.606482+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:23.606482+0000","last_peered":"2024-04-22T08:28:23.606482+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:23.606482+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":4858,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4796","reported_seq":4869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.568622+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:23.568622+0000","last_peered":"2024-04-22T08:28:23.568622+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:23.568622+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19629266,"num_objects":4794,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":19170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4910","reported_seq":5051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.530507+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:23.530507+0000","last_peered":"2024-04-22T08:28:23.530507+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:23.530507+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064676,"num_objects":4902,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":19599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":313458517,"num_objects":76539,"num_object_clones":0,"num_object_copies":153078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":76539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50141,"num_write_kb":200477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":628326400,"data_stored":628239425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76666,"ondisk_log_size":76666,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415020,"kb_used_data":271012,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":93956820,"statfs":{"total":96636764160,"available":96211783680,"internally_reserved":0,"allocated":277516288,"data_stored":277315273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147451599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920775,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159576,"kb_used_data":47568,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94212264,"statfs":{"total":96636764160,"available":96473358336,"internally_reserved":0,"allocated":48709632,"data_stored":48432079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":114683596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":21,"seq":90194313240,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455124,"kb_used_data":293132,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93916716,"statfs":{"total":96636764160,"available":96170717184,"internally_reserved":0,"allocated":300167168,"data_stored":299972615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":165867219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299954176,"data_stored":299950697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50294784,"data_stored":50214767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278077440,"data_stored":278073961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:24.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:25.830 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 73 20 78746 78726 4.21153 2.23438 0.0898669 0.018541 2024-04-22T08:28:25.831 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:25.832 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:25.893 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-04-22T08:28:24.161542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313458517,"num_objects":76539,"num_object_clones":0,"num_object_copies":153078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":76539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50141,"num_write_kb":200477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76666,"ondisk_log_size":76666,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1029720,"kb_used_data":611712,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282085800,"statfs":{"total":289910292480,"available":288855859200,"internally_reserved":0,"allocated":626393088,"data_stored":625719967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":428002414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,11,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32931840,"num_objects":8040,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-325,"num_objects_unfound":0,"num_objects_dirty":8040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":32160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":1331200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006344"},"pg_stats":[{"pgid":"2.f","version":"35'4847","reported_seq":4907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.581813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:23.581813+0000","last_peered":"2024-04-22T08:28:23.581813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:23.581813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4847,"num_write_kb":19388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1053},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4699","reported_seq":4811,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.503886+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:19.503886+0000","last_peered":"2024-04-22T08:28:19.503886+0000","last_clean":"2024-04-22T08:28:19.503886+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:19.503886+0000","last_undegraded":"2024-04-22T08:28:19.503886+0000","last_fullsized":"2024-04-22T08:28:19.503886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19222528,"num_objects":4693,"num_object_clones":0,"num_object_copies":9386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1755,"num_write_kb":7020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4668","reported_seq":4787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.452720+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:19.452720+0000","last_peered":"2024-04-22T08:28:19.452720+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:19.452720+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":4664,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":5980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4775","reported_seq":4802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:19.440560+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:19.440560+0000","last_peered":"2024-04-22T08:28:19.440560+0000","last_clean":"2024-04-22T08:28:19.440560+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:19.440560+0000","last_undegraded":"2024-04-22T08:28:19.440560+0000","last_fullsized":"2024-04-22T08:28:19.440560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19558400,"num_objects":4775,"num_object_clones":0,"num_object_copies":9550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4849","reported_seq":4981,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.566754+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:23.566754+0000","last_peered":"2024-04-22T08:28:23.566754+0000","last_clean":"2024-04-22T08:28:23.566754+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:23.566754+0000","last_undegraded":"2024-04-22T08:28:23.566754+0000","last_fullsized":"2024-04-22T08:28:23.566754+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":7764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4912","reported_seq":5071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.533436+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:23.533436+0000","last_peered":"2024-04-22T08:28:23.533436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:23.533436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20099072,"num_objects":4907,"num_object_clones":0,"num_object_copies":9814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1781,"num_write_kb":7124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1028},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4819","reported_seq":6138,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:23.600538+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:23.600538+0000","last_peered":"2024-04-22T08:28:23.600538+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:23.600538+0000","last_undegraded":"2024-04-22T08:28:23.600538+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19714048,"num_objects":4813,"num_object_clones":0,"num_object_copies":9626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1187,"num_write_kb":4748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1612},{"shards":"0,2","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4772","reported_seq":4913,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.585483+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:23.585483+0000","last_peered":"2024-04-22T08:28:23.585483+0000","last_clean":"2024-04-22T08:28:23.585483+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:23.585483+0000","last_undegraded":"2024-04-22T08:28:23.585483+0000","last_fullsized":"2024-04-22T08:28:23.585483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":4770,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":5680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'4851","reported_seq":4971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.458418+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:19.458418+0000","last_peered":"2024-04-22T08:28:19.458418+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:19.458418+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19604690,"num_objects":4788,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":19148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4814","reported_seq":4888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.562495+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:23.562495+0000","last_peered":"2024-04-22T08:28:23.562495+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:23.562495+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19686610,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":19228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1037},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4630","reported_seq":4772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.485673+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:19.485673+0000","last_peered":"2024-04-22T08:28:19.485673+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:19.485673+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18944000,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4705","reported_seq":4757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:19.470508+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:19.470508+0000","last_peered":"2024-04-22T08:28:19.470508+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:19.470508+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248338,"num_objects":4701,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4703,"num_write_kb":18799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4753","reported_seq":4829,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.580364+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:23.580364+0000","last_peered":"2024-04-22T08:28:23.580364+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:23.580364+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19460713,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4752,"num_write_kb":19005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1031},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4866","reported_seq":5037,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.606482+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:23.606482+0000","last_peered":"2024-04-22T08:28:23.606482+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:23.606482+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":4858,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4796","reported_seq":4869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.568622+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:23.568622+0000","last_peered":"2024-04-22T08:28:23.568622+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:23.568622+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19629266,"num_objects":4794,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":19170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4910","reported_seq":5051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.530507+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:23.530507+0000","last_peered":"2024-04-22T08:28:23.530507+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:23.530507+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064676,"num_objects":4902,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":19599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":313458517,"num_objects":76539,"num_object_clones":0,"num_object_copies":153078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":76539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50141,"num_write_kb":200477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":628326400,"data_stored":628239425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76666,"ondisk_log_size":76666,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415020,"kb_used_data":271012,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":93956820,"statfs":{"total":96636764160,"available":96211783680,"internally_reserved":0,"allocated":277516288,"data_stored":277315273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147451599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920775,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159576,"kb_used_data":47568,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94212264,"statfs":{"total":96636764160,"available":96473358336,"internally_reserved":0,"allocated":48709632,"data_stored":48432079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":114683596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":21,"seq":90194313240,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455124,"kb_used_data":293132,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93916716,"statfs":{"total":96636764160,"available":96170717184,"internally_reserved":0,"allocated":300167168,"data_stored":299972615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":165867219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299954176,"data_stored":299950697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50294784,"data_stored":50214767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278077440,"data_stored":278073961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:25.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:25.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'4847", 'reported_seq': 4907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.581813+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:23.581813+0000', 'last_peered': '2024-04-22T08:28:23.581813+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:23.581813+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4847, 'log_dups_size': 0, 'ondisk_log_size': 4847, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19853312, 'num_objects': 4847, 'num_object_clones': 0, 'num_object_copies': 9694, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4847, 'num_write_kb': 19388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1053}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'4668", 'reported_seq': 4787, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:19.452720+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:19.452720+0000', 'last_peered': '2024-04-22T08:28:19.452720+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:19.452720+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 4668, 'log_dups_size': 0, 'ondisk_log_size': 4668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19103744, 'num_objects': 4664, 'num_object_clones': 0, 'num_object_copies': 9328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1495, 'num_write_kb': 5980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'4912", 'reported_seq': 5071, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.533436+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:23.533436+0000', 'last_peered': '2024-04-22T08:28:23.533436+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:23.533436+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 4912, 'log_dups_size': 0, 'ondisk_log_size': 4912, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20099072, 'num_objects': 4907, 'num_object_clones': 0, 'num_object_copies': 9814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1781, 'num_write_kb': 7124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1028}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'4819", 'reported_seq': 6138, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:23.600538+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:23.600538+0000', 'last_peered': '2024-04-22T08:28:23.600538+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:23.600538+0000', 'last_undegraded': '2024-04-22T08:28:23.600538+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 4819, 'log_dups_size': 0, 'ondisk_log_size': 4819, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19714048, 'num_objects': 4813, 'num_object_clones': 0, 'num_object_copies': 9626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3201, 'num_objects_unfound': 0, 'num_objects_dirty': 4813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1187, 'num_write_kb': 4748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 580, 'num_bytes_recovered': 2375680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1612}, {'shards': '0,2', 'objects': 3201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'4851", 'reported_seq': 4971, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:19.458418+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:19.458418+0000', 'last_peered': '2024-04-22T08:28:19.458418+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:19.458418+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 4851, 'log_dups_size': 0, 'ondisk_log_size': 4851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19604690, 'num_objects': 4788, 'num_object_clones': 0, 'num_object_copies': 9576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4791, 'num_write_kb': 19148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'4814", 'reported_seq': 4888, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.562495+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:23.562495+0000', 'last_peered': '2024-04-22T08:28:23.562495+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:23.562495+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4814, 'log_dups_size': 0, 'ondisk_log_size': 4814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19686610, 'num_objects': 4808, 'num_object_clones': 0, 'num_object_copies': 9616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4812, 'num_write_kb': 19228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1037}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'4630", 'reported_seq': 4772, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:19.485673+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:19.485673+0000', 'last_peered': '2024-04-22T08:28:19.485673+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:19.485673+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 4630, 'log_dups_size': 0, 'ondisk_log_size': 4630, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18944000, 'num_objects': 4625, 'num_object_clones': 0, 'num_object_copies': 9250, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1084, 'num_write_kb': 4336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 875}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'4705", 'reported_seq': 4757, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:19.470508+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:19.470508+0000', 'last_peered': '2024-04-22T08:28:19.470508+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:19.470508+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4705, 'log_dups_size': 0, 'ondisk_log_size': 4705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19248338, 'num_objects': 4701, 'num_object_clones': 0, 'num_object_copies': 9402, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4703, 'num_write_kb': 18799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 898}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'4753", 'reported_seq': 4829, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.580364+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:23.580364+0000', 'last_peered': '2024-04-22T08:28:23.580364+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:23.580364+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4753, 'log_dups_size': 0, 'ondisk_log_size': 4753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19460713, 'num_objects': 4752, 'num_object_clones': 0, 'num_object_copies': 9504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4752, 'num_write_kb': 19005, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1031}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'4866", 'reported_seq': 5037, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.606482+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:23.606482+0000', 'last_peered': '2024-04-22T08:28:23.606482+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:23.606482+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 4866, 'log_dups_size': 0, 'ondisk_log_size': 4866, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19898368, 'num_objects': 4858, 'num_object_clones': 0, 'num_object_copies': 9716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1097, 'num_write_kb': 4388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1021}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'4796", 'reported_seq': 4869, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.568622+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:23.568622+0000', 'last_peered': '2024-04-22T08:28:23.568622+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:23.568622+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4796, 'log_dups_size': 0, 'ondisk_log_size': 4796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19629266, 'num_objects': 4794, 'num_object_clones': 0, 'num_object_copies': 9588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4794, 'num_write_kb': 19170, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1089}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:25.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'4910", 'reported_seq': 5051, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:23.530507+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:23.530507+0000', 'last_peered': '2024-04-22T08:28:23.530507+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:23.530507+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 4910, 'log_dups_size': 0, 'ondisk_log_size': 4910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20064676, 'num_objects': 4902, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4907, 'num_write_kb': 19599, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1021}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:25.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:26.830 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 74 20 79318 79298 4.18481 2.23438 0.00889356 0.0186551 2024-04-22T08:28:26.936 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:26.936 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/155288 objects degraded (26.920%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":141,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":77644,"data_bytes":317984597,"bytes_used":1064837120,"bytes_avail":288845455360,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":155288,"degraded_ratio":0.26920302921024164,"misplaced_objects":3201,"misplaced_total":155288,"misplaced_ratio":0.020613312039565197,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":74029,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"write_bytes_sec":2474363,"read_op_per_sec":0,"write_op_per_sec":604},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:26.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:26.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:27.811 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:27.830 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 75 20 79880 79860 4.15828 2.19531 0.0793411 0.0187794 2024-04-22T08:28:27.849 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-22T08:28:26.161989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317984597,"num_objects":77644,"num_object_clones":0,"num_object_copies":155288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":77644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51246,"num_write_kb":204897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77771,"ondisk_log_size":77771,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1039880,"kb_used_data":617776,"kb_used_omap":12,"kb_used_meta":422067,"kb_avail":282075640,"statfs":{"total":289910292480,"available":288845455360,"internally_reserved":0,"allocated":632602624,"data_stored":631929503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":432196718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,10,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29708288,"num_objects":7253,"num_object_clones":0,"num_object_copies":14506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-217,"num_objects_unfound":0,"num_objects_dirty":7253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7253,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006434"},"pg_stats":[{"pgid":"2.f","version":"35'4847","reported_seq":4907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.581813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:23.581813+0000","last_peered":"2024-04-22T08:28:23.581813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:23.581813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4847,"num_write_kb":19388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1053},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'4896","reported_seq":5008,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:24.514349+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:24.514349+0000","last_peered":"2024-04-22T08:28:24.514349+0000","last_clean":"2024-04-22T08:28:24.514349+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:24.514349+0000","last_undegraded":"2024-04-22T08:28:24.514349+0000","last_fullsized":"2024-04-22T08:28:24.514349+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20029440,"num_objects":4890,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1952,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4848","reported_seq":4967,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:24.475197+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:24.475197+0000","last_peered":"2024-04-22T08:28:24.475197+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:24.475197+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19841024,"num_objects":4844,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1675,"num_write_kb":6700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'4923","reported_seq":4950,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:24.391750+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:24.391750+0000","last_peered":"2024-04-22T08:28:24.391750+0000","last_clean":"2024-04-22T08:28:24.391750+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:24.391750+0000","last_undegraded":"2024-04-22T08:28:24.391750+0000","last_fullsized":"2024-04-22T08:28:24.391750+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164608,"num_objects":4923,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'4849","reported_seq":4981,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.566754+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:23.566754+0000","last_peered":"2024-04-22T08:28:23.566754+0000","last_clean":"2024-04-22T08:28:23.566754+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:23.566754+0000","last_undegraded":"2024-04-22T08:28:23.566754+0000","last_fullsized":"2024-04-22T08:28:23.566754+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":7764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'4912","reported_seq":5071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.533436+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:23.533436+0000","last_peered":"2024-04-22T08:28:23.533436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:23.533436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20099072,"num_objects":4907,"num_object_clones":0,"num_object_copies":9814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1781,"num_write_kb":7124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1028},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4819","reported_seq":6138,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:23.600538+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:23.600538+0000","last_peered":"2024-04-22T08:28:23.600538+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:23.600538+0000","last_undegraded":"2024-04-22T08:28:23.600538+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19714048,"num_objects":4813,"num_object_clones":0,"num_object_copies":9626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1187,"num_write_kb":4748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1612},{"shards":"0,2","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4772","reported_seq":4913,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:23.585483+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:23.585483+0000","last_peered":"2024-04-22T08:28:23.585483+0000","last_clean":"2024-04-22T08:28:23.585483+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:23.585483+0000","last_undegraded":"2024-04-22T08:28:23.585483+0000","last_fullsized":"2024-04-22T08:28:23.585483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":4770,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":5680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5040","reported_seq":5158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:24.466546+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:24.466546+0000","last_peered":"2024-04-22T08:28:24.466546+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:24.466546+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20378834,"num_objects":4977,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":19904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'4814","reported_seq":4888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.562495+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:23.562495+0000","last_peered":"2024-04-22T08:28:23.562495+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:23.562495+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19686610,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":19228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1037},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'4835","reported_seq":4977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:24.501834+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:24.501834+0000","last_peered":"2024-04-22T08:28:24.501834+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:24.501834+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19783680,"num_objects":4830,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":5156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'4891","reported_seq":4943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:24.508890+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:24.508890+0000","last_peered":"2024-04-22T08:28:24.508890+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:24.508890+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010194,"num_objects":4887,"num_object_clones":0,"num_object_copies":9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":19543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1084}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4753","reported_seq":4829,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.580364+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:23.580364+0000","last_peered":"2024-04-22T08:28:23.580364+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:23.580364+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19460713,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4752,"num_write_kb":19005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1031},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'4866","reported_seq":5037,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.606482+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:23.606482+0000","last_peered":"2024-04-22T08:28:23.606482+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:23.606482+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":4858,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4796","reported_seq":4869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.568622+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:23.568622+0000","last_peered":"2024-04-22T08:28:23.568622+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:23.568622+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19629266,"num_objects":4794,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":19170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'4910","reported_seq":5051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:23.530507+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:23.530507+0000","last_peered":"2024-04-22T08:28:23.530507+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:23.530507+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064676,"num_objects":4902,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":19599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1021},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":317984597,"num_objects":77644,"num_object_clones":0,"num_object_copies":155288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3201,"num_objects_unfound":0,"num_objects_dirty":77644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51246,"num_write_kb":204897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":2392681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":634556416,"data_stored":634469441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77771,"ondisk_log_size":77771,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117013,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425180,"kb_used_data":277076,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93946660,"statfs":{"total":96636764160,"available":96201379840,"internally_reserved":0,"allocated":283725824,"data_stored":283524809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151645903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.9910000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920775,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159576,"kb_used_data":47568,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94212264,"statfs":{"total":96636764160,"available":96473358336,"internally_reserved":0,"allocated":48709632,"data_stored":48432079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":114683596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":21,"seq":90194313240,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455124,"kb_used_data":293132,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93916716,"statfs":{"total":96636764160,"available":96170717184,"internally_reserved":0,"allocated":300167168,"data_stored":299972615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":165867219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299954176,"data_stored":299950697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50294784,"data_stored":50214767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284307456,"data_stored":284303977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:28.831 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 76 20 80489 80469 4.13487 2.37891 0.0120762 0.0188853 2024-04-22T08:28:29.831 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 77 20 81097 81077 4.11202 2.375 0.0106769 0.0189907 2024-04-22T08:28:30.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 78 20 81687 81667 4.08881 2.30469 0.0135097 0.0191001 2024-04-22T08:28:30.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:31.778 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:31.778 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:31.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 79 20 82265 82245 4.06563 2.25781 0.0141513 0.0192081 2024-04-22T08:28:31.883 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-04-22T08:28:30.163661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330026837,"num_objects":80584,"num_object_clones":0,"num_object_copies":161168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":80584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54186,"num_write_kb":216657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":2921065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80711,"ondisk_log_size":80711,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1051672,"kb_used_data":641856,"kb_used_omap":12,"kb_used_meta":409779,"kb_avail":282063848,"statfs":{"total":289910292480,"available":288833380352,"internally_reserved":0,"allocated":657260544,"data_stored":656587423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":419613806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,7,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37015552,"num_objects":9037,"num_object_clones":0,"num_object_copies":18074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-346,"num_objects_unfound":0,"num_objects_dirty":9037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":36148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006498"},"pg_stats":[{"pgid":"2.f","version":"35'5025","reported_seq":5085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.631755+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:28.631755+0000","last_peered":"2024-04-22T08:28:28.631755+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:28.631755+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20582400,"num_objects":5025,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":20100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1231},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5057","reported_seq":5169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.474877+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:29.474877+0000","last_peered":"2024-04-22T08:28:29.474877+0000","last_clean":"2024-04-22T08:28:29.474877+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:29.474877+0000","last_undegraded":"2024-04-22T08:28:29.474877+0000","last_fullsized":"2024-04-22T08:28:29.474877+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20688896,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2113,"num_write_kb":8452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5052","reported_seq":5172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.512378+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:29.512378+0000","last_peered":"2024-04-22T08:28:29.512378+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:29.512378+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20676608,"num_objects":5048,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1879,"num_write_kb":7516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5122","reported_seq":5149,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.537887+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:29.537887+0000","last_peered":"2024-04-22T08:28:29.537887+0000","last_clean":"2024-04-22T08:28:29.537887+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:29.537887+0000","last_undegraded":"2024-04-22T08:28:29.537887+0000","last_fullsized":"2024-04-22T08:28:29.537887+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20979712,"num_objects":5122,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5034","reported_seq":5167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:28.606083+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:28.606083+0000","last_peered":"2024-04-22T08:28:28.606083+0000","last_clean":"2024-04-22T08:28:28.606083+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:28.606083+0000","last_undegraded":"2024-04-22T08:28:28.606083+0000","last_fullsized":"2024-04-22T08:28:28.606083+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20590592,"num_objects":5027,"num_object_clones":0,"num_object_copies":10054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":8504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5096","reported_seq":5255,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.617678+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:28.617678+0000","last_peered":"2024-04-22T08:28:28.617678+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:28.617678+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20852736,"num_objects":5091,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1965,"num_write_kb":7860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1212},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4968","reported_seq":6545,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:28.634062+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:28.634062+0000","last_peered":"2024-04-22T08:28:28.634062+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:28.634062+0000","last_undegraded":"2024-04-22T08:28:28.634062+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20324352,"num_objects":4962,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1336,"num_write_kb":5344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":709,"num_bytes_recovered":2904064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1890},{"shards":"0,2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4958","reported_seq":5099,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:28.640643+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:28.640643+0000","last_peered":"2024-04-22T08:28:28.640643+0000","last_clean":"2024-04-22T08:28:28.640643+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:28.640643+0000","last_undegraded":"2024-04-22T08:28:28.640643+0000","last_fullsized":"2024-04-22T08:28:28.640643+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":6424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5224","reported_seq":5344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.503640+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:29.503640+0000","last_peered":"2024-04-22T08:28:29.503640+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:29.503640+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21132498,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":20640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5007","reported_seq":5081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.611336+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:28.611336+0000","last_peered":"2024-04-22T08:28:28.611336+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:28.611336+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20477138,"num_objects":5001,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1230},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5025","reported_seq":5166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.515253+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:29.515253+0000","last_peered":"2024-04-22T08:28:29.515253+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:29.515253+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":5916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5080","reported_seq":5132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.524673+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:29.524673+0000","last_peered":"2024-04-22T08:28:29.524673+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:29.524673+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5080,"log_dups_size":0,"ondisk_log_size":5080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20784338,"num_objects":5076,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5078,"num_write_kb":20299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4941","reported_seq":5017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.631730+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:28.631730+0000","last_peered":"2024-04-22T08:28:28.631730+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:28.631730+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230761,"num_objects":4940,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":19757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1219},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5044","reported_seq":5216,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.626323+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:28.626323+0000","last_peered":"2024-04-22T08:28:28.626323+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:28.626323+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":5036,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":5100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1199},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4971","reported_seq":5044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.629746+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:28.629746+0000","last_peered":"2024-04-22T08:28:28.629746+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:28.629746+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20346066,"num_objects":4969,"num_object_clones":0,"num_object_copies":9938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4969,"num_write_kb":19870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1264},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5107","reported_seq":5247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.640098+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:28.640098+0000","last_peered":"2024-04-22T08:28:28.640098+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:28.640098+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20871588,"num_objects":5099,"num_object_clones":0,"num_object_copies":10198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":20387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1218},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":330026837,"num_objects":80584,"num_object_clones":0,"num_object_copies":161168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":80584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54186,"num_write_kb":216657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":2921065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":659218432,"data_stored":659131457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80711,"ondisk_log_size":80711,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418908,"kb_used_data":283092,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93952932,"statfs":{"total":96636764160,"available":96207802368,"internally_reserved":0,"allocated":289886208,"data_stored":289685193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":2.6499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":1,"up_from":33,"seq":141733920776,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160848,"kb_used_data":57032,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94210992,"statfs":{"total":96636764160,"available":96472055808,"internally_reserved":0,"allocated":58400768,"data_stored":58123215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":106294988},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":0,"up_from":21,"seq":90194313241,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471916,"kb_used_data":301732,"kb_used_omap":4,"kb_used_meta":170171,"kb_avail":93899924,"statfs":{"total":96636764160,"available":96153522176,"internally_reserved":0,"allocated":308973568,"data_stored":308779015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":174255827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.245},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308858880,"data_stored":308855401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59936768,"data_stored":59856751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290422784,"data_stored":290419305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:31.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:32.763 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:32.764 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-04-22T08:28:32.164150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330026837,"num_objects":80584,"num_object_clones":0,"num_object_copies":161168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":80584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54186,"num_write_kb":216657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":2921065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80711,"ondisk_log_size":80711,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1073444,"kb_used_data":651340,"kb_used_omap":12,"kb_used_meta":422067,"kb_avail":282042076,"statfs":{"total":289910292480,"available":288811085824,"internally_reserved":0,"allocated":666972160,"data_stored":666299039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":432196718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,8,11,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24354816,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006561"},"pg_stats":[{"pgid":"2.f","version":"35'5025","reported_seq":5085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.631755+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:28.631755+0000","last_peered":"2024-04-22T08:28:28.631755+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:28.631755+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20582400,"num_objects":5025,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":20100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1231},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5057","reported_seq":5169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.474877+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:29.474877+0000","last_peered":"2024-04-22T08:28:29.474877+0000","last_clean":"2024-04-22T08:28:29.474877+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:29.474877+0000","last_undegraded":"2024-04-22T08:28:29.474877+0000","last_fullsized":"2024-04-22T08:28:29.474877+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20688896,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2113,"num_write_kb":8452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5052","reported_seq":5172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.512378+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:29.512378+0000","last_peered":"2024-04-22T08:28:29.512378+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:29.512378+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20676608,"num_objects":5048,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1879,"num_write_kb":7516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5122","reported_seq":5149,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.537887+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:29.537887+0000","last_peered":"2024-04-22T08:28:29.537887+0000","last_clean":"2024-04-22T08:28:29.537887+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:29.537887+0000","last_undegraded":"2024-04-22T08:28:29.537887+0000","last_fullsized":"2024-04-22T08:28:29.537887+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20979712,"num_objects":5122,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5034","reported_seq":5167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:28.606083+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:28.606083+0000","last_peered":"2024-04-22T08:28:28.606083+0000","last_clean":"2024-04-22T08:28:28.606083+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:28.606083+0000","last_undegraded":"2024-04-22T08:28:28.606083+0000","last_fullsized":"2024-04-22T08:28:28.606083+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20590592,"num_objects":5027,"num_object_clones":0,"num_object_copies":10054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":8504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5096","reported_seq":5255,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.617678+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:28.617678+0000","last_peered":"2024-04-22T08:28:28.617678+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:28.617678+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20852736,"num_objects":5091,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1965,"num_write_kb":7860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1212},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'4968","reported_seq":6545,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:28.634062+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:28.634062+0000","last_peered":"2024-04-22T08:28:28.634062+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:28.634062+0000","last_undegraded":"2024-04-22T08:28:28.634062+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20324352,"num_objects":4962,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1336,"num_write_kb":5344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":709,"num_bytes_recovered":2904064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1890},{"shards":"0,2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'4958","reported_seq":5099,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:28.640643+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:28.640643+0000","last_peered":"2024-04-22T08:28:28.640643+0000","last_clean":"2024-04-22T08:28:28.640643+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:28.640643+0000","last_undegraded":"2024-04-22T08:28:28.640643+0000","last_fullsized":"2024-04-22T08:28:28.640643+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":6424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5224","reported_seq":5344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.503640+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:29.503640+0000","last_peered":"2024-04-22T08:28:29.503640+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:29.503640+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21132498,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":20640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5007","reported_seq":5081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.611336+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:28.611336+0000","last_peered":"2024-04-22T08:28:28.611336+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:28.611336+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20477138,"num_objects":5001,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1230},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5025","reported_seq":5166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.515253+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:29.515253+0000","last_peered":"2024-04-22T08:28:29.515253+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:29.515253+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":5916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5080","reported_seq":5132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.524673+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:29.524673+0000","last_peered":"2024-04-22T08:28:29.524673+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:29.524673+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5080,"log_dups_size":0,"ondisk_log_size":5080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20784338,"num_objects":5076,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5078,"num_write_kb":20299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'4941","reported_seq":5017,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.631730+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:28.631730+0000","last_peered":"2024-04-22T08:28:28.631730+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:28.631730+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230761,"num_objects":4940,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":19757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1219},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5044","reported_seq":5216,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.626323+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:28.626323+0000","last_peered":"2024-04-22T08:28:28.626323+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:28.626323+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":5036,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":5100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1199},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'4971","reported_seq":5044,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.629746+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:28.629746+0000","last_peered":"2024-04-22T08:28:28.629746+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:28.629746+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20346066,"num_objects":4969,"num_object_clones":0,"num_object_copies":9938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4969,"num_write_kb":19870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1264},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5107","reported_seq":5247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:28.640098+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:28.640098+0000","last_peered":"2024-04-22T08:28:28.640098+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:28.640098+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20871588,"num_objects":5099,"num_object_clones":0,"num_object_copies":10198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":20387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1218},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":330026837,"num_objects":80584,"num_object_clones":0,"num_object_copies":161168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":80584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54186,"num_write_kb":216657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":2921065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668987392,"data_stored":668900417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80711,"ondisk_log_size":80711,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418908,"kb_used_data":283092,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93952932,"statfs":{"total":96636764160,"available":96207802368,"internally_reserved":0,"allocated":289886208,"data_stored":289685193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":2.6499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":1,"up_from":33,"seq":141733920777,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182620,"kb_used_data":66516,"kb_used_omap":4,"kb_used_meta":116091,"kb_avail":94189220,"statfs":{"total":96636764160,"available":96449761280,"internally_reserved":0,"allocated":68112384,"data_stored":67834831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":118877900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5510000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313241,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471916,"kb_used_data":301732,"kb_used_omap":4,"kb_used_meta":170171,"kb_avail":93899924,"statfs":{"total":96636764160,"available":96153522176,"internally_reserved":0,"allocated":308973568,"data_stored":308779015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":174255827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.245},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308858880,"data_stored":308855401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69705728,"data_stored":69625711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290422784,"data_stored":290419305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:32.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:32.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'5025", 'reported_seq': 5085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.631755+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:28.631755+0000', 'last_peered': '2024-04-22T08:28:28.631755+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:28.631755+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5025, 'log_dups_size': 0, 'ondisk_log_size': 5025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20582400, 'num_objects': 5025, 'num_object_clones': 0, 'num_object_copies': 10050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5025, 'num_write_kb': 20100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1231}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'5052", 'reported_seq': 5172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:29.512378+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:29.512378+0000', 'last_peered': '2024-04-22T08:28:29.512378+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:29.512378+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 5052, 'log_dups_size': 0, 'ondisk_log_size': 5052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20676608, 'num_objects': 5048, 'num_object_clones': 0, 'num_object_copies': 10096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1879, 'num_write_kb': 7516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 1293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'5096", 'reported_seq': 5255, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.617678+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:28.617678+0000', 'last_peered': '2024-04-22T08:28:28.617678+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:28.617678+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 5096, 'log_dups_size': 0, 'ondisk_log_size': 5096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20852736, 'num_objects': 5091, 'num_object_clones': 0, 'num_object_copies': 10182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1965, 'num_write_kb': 7860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1212}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'4968", 'reported_seq': 6545, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:28.634062+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:28.634062+0000', 'last_peered': '2024-04-22T08:28:28.634062+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:28.634062+0000', 'last_undegraded': '2024-04-22T08:28:28.634062+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 4968, 'log_dups_size': 0, 'ondisk_log_size': 4968, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20324352, 'num_objects': 4962, 'num_object_clones': 0, 'num_object_copies': 9924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3072, 'num_objects_unfound': 0, 'num_objects_dirty': 4962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1336, 'num_write_kb': 5344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 709, 'num_bytes_recovered': 2904064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1890}, {'shards': '0,2', 'objects': 3072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'5224", 'reported_seq': 5344, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:29.503640+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:29.503640+0000', 'last_peered': '2024-04-22T08:28:29.503640+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:29.503640+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 5224, 'log_dups_size': 0, 'ondisk_log_size': 5224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21132498, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 10322, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5164, 'num_write_kb': 20640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 1273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:32.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'5007", 'reported_seq': 5081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.611336+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:28.611336+0000', 'last_peered': '2024-04-22T08:28:28.611336+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:28.611336+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5007, 'log_dups_size': 0, 'ondisk_log_size': 5007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20477138, 'num_objects': 5001, 'num_object_clones': 0, 'num_object_copies': 10002, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 20000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1230}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'5025", 'reported_seq': 5166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:29.515253+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:29.515253+0000', 'last_peered': '2024-04-22T08:28:29.515253+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:29.515253+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 5025, 'log_dups_size': 0, 'ondisk_log_size': 5025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20561920, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 10040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1479, 'num_write_kb': 5916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 1270}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'5080", 'reported_seq': 5132, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:29.524673+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:29.524673+0000', 'last_peered': '2024-04-22T08:28:29.524673+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:29.524673+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5080, 'log_dups_size': 0, 'ondisk_log_size': 5080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20784338, 'num_objects': 5076, 'num_object_clones': 0, 'num_object_copies': 10152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5078, 'num_write_kb': 20299, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 1273}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'4941", 'reported_seq': 5017, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.631730+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:28.631730+0000', 'last_peered': '2024-04-22T08:28:28.631730+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:28.631730+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4941, 'log_dups_size': 0, 'ondisk_log_size': 4941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20230761, 'num_objects': 4940, 'num_object_clones': 0, 'num_object_copies': 9880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4940, 'num_write_kb': 19757, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1219}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'5044", 'reported_seq': 5216, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.626323+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:28.626323+0000', 'last_peered': '2024-04-22T08:28:28.626323+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:28.626323+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 5044, 'log_dups_size': 0, 'ondisk_log_size': 5044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20627456, 'num_objects': 5036, 'num_object_clones': 0, 'num_object_copies': 10072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1275, 'num_write_kb': 5100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1199}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:32.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'4971", 'reported_seq': 5044, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.629746+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:28.629746+0000', 'last_peered': '2024-04-22T08:28:28.629746+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:28.629746+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 4971, 'log_dups_size': 0, 'ondisk_log_size': 4971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20346066, 'num_objects': 4969, 'num_object_clones': 0, 'num_object_copies': 9938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4969, 'num_write_kb': 19870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1264}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.813 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:32.813 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'5107", 'reported_seq': 5247, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:28.640098+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:28.640098+0000', 'last_peered': '2024-04-22T08:28:28.640098+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:28.640098+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 5107, 'log_dups_size': 0, 'ondisk_log_size': 5107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20871588, 'num_objects': 5099, 'num_object_clones': 0, 'num_object_copies': 10198, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5104, 'num_write_kb': 20387, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1218}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:32.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:32.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:28:32.835135+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0193193 2024-04-22T08:28:32.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:28:32.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 80 20 82826 82806 4.0422 2.19141 0.0748936 0.0193193 2024-04-22T08:28:33.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 81 20 83390 83370 4.0195 2.20312 0.0400665 0.0194273 2024-04-22T08:28:34.016 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:34.016 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/161168 objects degraded (25.938%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":148,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":80584,"data_bytes":330026837,"bytes_used":1099206656,"bytes_avail":288811085824,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":161168,"degraded_ratio":0.2593815149409312,"misplaced_objects":3072,"misplaced_total":161168,"misplaced_ratio":0.019060855753003078,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":81875,"recovering_keys_per_sec":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":2028458,"read_op_per_sec":0,"write_op_per_sec":495},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:34.099 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:34.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:34.827 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:34.827 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:34.832 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 82 20 83949 83929 3.99711 2.18359 0.0340692 0.0195362 2024-04-22T08:28:34.863 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-22T08:28:34.164609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337542997,"num_objects":82419,"num_object_clones":0,"num_object_copies":164838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":82419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56021,"num_write_kb":223997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82546,"ondisk_log_size":82546,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1087908,"kb_used_data":660044,"kb_used_omap":12,"kb_used_meta":427827,"kb_avail":282027612,"statfs":{"total":289910292480,"available":288796274688,"internally_reserved":0,"allocated":675885056,"data_stored":675211935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":438094958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,9,6,11,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31870976,"num_objects":7781,"num_object_clones":0,"num_object_copies":15562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-364,"num_objects_unfound":0,"num_objects_dirty":7781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":31124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006529"},"pg_stats":[{"pgid":"2.f","version":"35'5227","reported_seq":5287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.657759+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:33.657759+0000","last_peered":"2024-04-22T08:28:33.657759+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:33.657759+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21409792,"num_objects":5227,"num_object_clones":0,"num_object_copies":10454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1433},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5057","reported_seq":5169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.474877+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:29.474877+0000","last_peered":"2024-04-22T08:28:29.474877+0000","last_clean":"2024-04-22T08:28:29.474877+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:29.474877+0000","last_undegraded":"2024-04-22T08:28:29.474877+0000","last_fullsized":"2024-04-22T08:28:29.474877+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20688896,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2113,"num_write_kb":8452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5052","reported_seq":5172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.512378+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:29.512378+0000","last_peered":"2024-04-22T08:28:29.512378+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:29.512378+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20676608,"num_objects":5048,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1879,"num_write_kb":7516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5122","reported_seq":5149,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:29.537887+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:29.537887+0000","last_peered":"2024-04-22T08:28:29.537887+0000","last_clean":"2024-04-22T08:28:29.537887+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:29.537887+0000","last_undegraded":"2024-04-22T08:28:29.537887+0000","last_fullsized":"2024-04-22T08:28:29.537887+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20979712,"num_objects":5122,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5201","reported_seq":5336,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.647367+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:33.647367+0000","last_peered":"2024-04-22T08:28:33.647367+0000","last_clean":"2024-04-22T08:28:33.647367+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:33.647367+0000","last_undegraded":"2024-04-22T08:28:33.647367+0000","last_fullsized":"2024-04-22T08:28:33.647367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":5194,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5280","reported_seq":5436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.565951+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:33.565951+0000","last_peered":"2024-04-22T08:28:33.565951+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:33.565951+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21606400,"num_objects":5275,"num_object_clones":0,"num_object_copies":10550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2149,"num_write_kb":8596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1396},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5145","reported_seq":6970,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:33.656479+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:33.656479+0000","last_peered":"2024-04-22T08:28:33.656479+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:33.656479+0000","last_undegraded":"2024-04-22T08:28:33.656479+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21049344,"num_objects":5139,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2191},{"shards":"0,2","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5126","reported_seq":5267,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.588226+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:33.588226+0000","last_peered":"2024-04-22T08:28:33.588226+0000","last_clean":"2024-04-22T08:28:33.588226+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:33.588226+0000","last_undegraded":"2024-04-22T08:28:33.588226+0000","last_fullsized":"2024-04-22T08:28:33.588226+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20987904,"num_objects":5124,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":7096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5224","reported_seq":5344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.503640+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:29.503640+0000","last_peered":"2024-04-22T08:28:29.503640+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:29.503640+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21132498,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":20640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5199","reported_seq":5270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.678338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:33.678338+0000","last_peered":"2024-04-22T08:28:33.678338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:33.678338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263570,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":20768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1422},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5025","reported_seq":5166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.515253+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:29.515253+0000","last_peered":"2024-04-22T08:28:29.515253+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:29.515253+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":5916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5080","reported_seq":5132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:29.524673+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:29.524673+0000","last_peered":"2024-04-22T08:28:29.524673+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:29.524673+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5080,"log_dups_size":0,"ondisk_log_size":5080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20784338,"num_objects":5076,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5078,"num_write_kb":20299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5111","reported_seq":5187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.595848+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:33.595848+0000","last_peered":"2024-04-22T08:28:33.595848+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:33.595848+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927081,"num_objects":5110,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1389},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5230","reported_seq":5402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.631996+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:33.631996+0000","last_peered":"2024-04-22T08:28:33.631996+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:33.631996+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21389312,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":5844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1385},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5171","reported_seq":5244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.639110+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:33.639110+0000","last_peered":"2024-04-22T08:28:33.639110+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:33.639110+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21165266,"num_objects":5169,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":20670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1464},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5296","reported_seq":5437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.653233+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:33.653233+0000","last_peered":"2024-04-22T08:28:33.653233+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:33.653233+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21645732,"num_objects":5288,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5293,"num_write_kb":21143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1407},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":337542997,"num_objects":82419,"num_object_clones":0,"num_object_copies":164838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":82419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56021,"num_write_kb":223997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678068224,"data_stored":677981249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82546,"ondisk_log_size":82546,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418908,"kb_used_data":283092,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93952932,"statfs":{"total":96636764160,"available":96207802368,"internally_reserved":0,"allocated":289886208,"data_stored":289685193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":2.6499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":1,"up_from":33,"seq":141733920777,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182620,"kb_used_data":66516,"kb_used_omap":4,"kb_used_meta":116091,"kb_avail":94189220,"statfs":{"total":96636764160,"available":96449761280,"internally_reserved":0,"allocated":68112384,"data_stored":67834831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":118877900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5510000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313242,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486380,"kb_used_data":310436,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93885460,"statfs":{"total":96636764160,"available":96138711040,"internally_reserved":0,"allocated":317886464,"data_stored":317691911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317939712,"data_stored":317936233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69705728,"data_stored":69625711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290422784,"data_stored":290419305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:35.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 83 20 84575 84555 3.97841 2.44531 0.0129714 0.0196289 2024-04-22T08:28:36.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 84 20 85177 85157 3.95905 2.35156 0.050677 0.0197236 2024-04-22T08:28:37.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 85 20 85642 85622 3.93384 1.81641 0.027394 0.0198497 2024-04-22T08:28:37.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:38.623 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:38.623 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:38.671 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-22T08:28:38.165613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":341847893,"num_objects":83470,"num_object_clones":0,"num_object_copies":166940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":83470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57072,"num_write_kb":228201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83597,"ondisk_log_size":83597,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1122672,"kb_used_data":676760,"kb_used_omap":12,"kb_used_meta":445875,"kb_avail":281992848,"statfs":{"total":289910292480,"available":288760676352,"internally_reserved":0,"allocated":693002240,"data_stored":692329119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":456576110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,7,6,11,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23863296,"num_objects":5826,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-253,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":23304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":1036288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003624"},"pg_stats":[{"pgid":"2.f","version":"35'5227","reported_seq":5287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.657759+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:33.657759+0000","last_peered":"2024-04-22T08:28:33.657759+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:33.657759+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21409792,"num_objects":5227,"num_object_clones":0,"num_object_copies":10454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1433},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5245","reported_seq":5357,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:34.484185+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:34.484185+0000","last_peered":"2024-04-22T08:28:34.484185+0000","last_clean":"2024-04-22T08:28:34.484185+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:34.484185+0000","last_undegraded":"2024-04-22T08:28:34.484185+0000","last_fullsized":"2024-04-22T08:28:34.484185+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21458944,"num_objects":5239,"num_object_clones":0,"num_object_copies":10478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":9204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5224","reported_seq":5343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.529065+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:34.529065+0000","last_peered":"2024-04-22T08:28:34.529065+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:34.529065+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21381120,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5286","reported_seq":5313,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:34.370095+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:34.370095+0000","last_peered":"2024-04-22T08:28:34.370095+0000","last_clean":"2024-04-22T08:28:34.370095+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:34.370095+0000","last_undegraded":"2024-04-22T08:28:34.370095+0000","last_fullsized":"2024-04-22T08:28:34.370095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21651456,"num_objects":5286,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5286,"num_write_kb":21144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5201","reported_seq":5336,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.647367+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:33.647367+0000","last_peered":"2024-04-22T08:28:33.647367+0000","last_clean":"2024-04-22T08:28:33.647367+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:33.647367+0000","last_undegraded":"2024-04-22T08:28:33.647367+0000","last_fullsized":"2024-04-22T08:28:33.647367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":5194,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5280","reported_seq":5436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.565951+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:33.565951+0000","last_peered":"2024-04-22T08:28:33.565951+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:33.565951+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21606400,"num_objects":5275,"num_object_clones":0,"num_object_copies":10550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2149,"num_write_kb":8596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1396},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5145","reported_seq":6970,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:33.656479+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:33.656479+0000","last_peered":"2024-04-22T08:28:33.656479+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:33.656479+0000","last_undegraded":"2024-04-22T08:28:33.656479+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21049344,"num_objects":5139,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2191},{"shards":"0,2","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5126","reported_seq":5267,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.588226+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:33.588226+0000","last_peered":"2024-04-22T08:28:33.588226+0000","last_clean":"2024-04-22T08:28:33.588226+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:33.588226+0000","last_undegraded":"2024-04-22T08:28:33.588226+0000","last_fullsized":"2024-04-22T08:28:33.588226+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20987904,"num_objects":5124,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":7096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5383","reported_seq":5503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.484791+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:34.484791+0000","last_peered":"2024-04-22T08:28:34.484791+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:34.484791+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21783762,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5323,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5199","reported_seq":5270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.678338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:33.678338+0000","last_peered":"2024-04-22T08:28:33.678338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:33.678338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263570,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":20768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1422},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5209","reported_seq":5352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.461097+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:34.461097+0000","last_peered":"2024-04-22T08:28:34.461097+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:34.461097+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21315584,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1663,"num_write_kb":6652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5264","reported_seq":5316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.503971+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:34.503971+0000","last_peered":"2024-04-22T08:28:34.503971+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:34.503971+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21538002,"num_objects":5260,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":21035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1457}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5111","reported_seq":5187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.595848+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:33.595848+0000","last_peered":"2024-04-22T08:28:33.595848+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:33.595848+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927081,"num_objects":5110,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1389},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5230","reported_seq":5402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.631996+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:33.631996+0000","last_peered":"2024-04-22T08:28:33.631996+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:33.631996+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21389312,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":5844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1385},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5171","reported_seq":5244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.639110+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:33.639110+0000","last_peered":"2024-04-22T08:28:33.639110+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:33.639110+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21165266,"num_objects":5169,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":20670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1464},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5296","reported_seq":5437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.653233+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:33.653233+0000","last_peered":"2024-04-22T08:28:33.653233+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:33.653233+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21645732,"num_objects":5288,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5293,"num_write_kb":21143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1407},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":341847893,"num_objects":83470,"num_object_clones":0,"num_object_copies":166940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":83470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57072,"num_write_kb":228201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693227520,"data_stored":693140545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83597,"ondisk_log_size":83597,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450872,"kb_used_data":288816,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93920968,"statfs":{"total":96636764160,"available":96175071232,"internally_reserved":0,"allocated":295747584,"data_stored":295546569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165932751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":3.9780000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6029999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920778,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185420,"kb_used_data":77508,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94186420,"statfs":{"total":96636764160,"available":96446894080,"internally_reserved":0,"allocated":79368192,"data_stored":79090639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":110489292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313242,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486380,"kb_used_data":310436,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93885460,"statfs":{"total":96636764160,"available":96138711040,"internally_reserved":0,"allocated":317886464,"data_stored":317691911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317939712,"data_stored":317936233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79192064,"data_stored":79112047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296095744,"data_stored":296092265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:38.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:38.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 86 20 86183 86163 3.91267 2.11328 0.0316596 0.0199569 2024-04-22T08:28:39.440 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:39.440 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:39.509 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-22T08:28:38.165613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":341847893,"num_objects":83470,"num_object_clones":0,"num_object_copies":166940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":83470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57072,"num_write_kb":228201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83597,"ondisk_log_size":83597,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1122672,"kb_used_data":676760,"kb_used_omap":12,"kb_used_meta":445875,"kb_avail":281992848,"statfs":{"total":289910292480,"available":288760676352,"internally_reserved":0,"allocated":693002240,"data_stored":692329119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":456576110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,7,6,11,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23863296,"num_objects":5826,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-253,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":23304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":1036288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003624"},"pg_stats":[{"pgid":"2.f","version":"35'5227","reported_seq":5287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.657759+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:33.657759+0000","last_peered":"2024-04-22T08:28:33.657759+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:33.657759+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21409792,"num_objects":5227,"num_object_clones":0,"num_object_copies":10454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1433},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5245","reported_seq":5357,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:34.484185+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:34.484185+0000","last_peered":"2024-04-22T08:28:34.484185+0000","last_clean":"2024-04-22T08:28:34.484185+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:34.484185+0000","last_undegraded":"2024-04-22T08:28:34.484185+0000","last_fullsized":"2024-04-22T08:28:34.484185+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21458944,"num_objects":5239,"num_object_clones":0,"num_object_copies":10478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":9204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5224","reported_seq":5343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.529065+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:34.529065+0000","last_peered":"2024-04-22T08:28:34.529065+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:34.529065+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21381120,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5286","reported_seq":5313,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:34.370095+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:34.370095+0000","last_peered":"2024-04-22T08:28:34.370095+0000","last_clean":"2024-04-22T08:28:34.370095+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:34.370095+0000","last_undegraded":"2024-04-22T08:28:34.370095+0000","last_fullsized":"2024-04-22T08:28:34.370095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21651456,"num_objects":5286,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5286,"num_write_kb":21144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5201","reported_seq":5336,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.647367+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:33.647367+0000","last_peered":"2024-04-22T08:28:33.647367+0000","last_clean":"2024-04-22T08:28:33.647367+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:33.647367+0000","last_undegraded":"2024-04-22T08:28:33.647367+0000","last_fullsized":"2024-04-22T08:28:33.647367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":5194,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5280","reported_seq":5436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.565951+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:33.565951+0000","last_peered":"2024-04-22T08:28:33.565951+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:33.565951+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21606400,"num_objects":5275,"num_object_clones":0,"num_object_copies":10550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2149,"num_write_kb":8596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1396},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5145","reported_seq":6970,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:33.656479+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:33.656479+0000","last_peered":"2024-04-22T08:28:33.656479+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:33.656479+0000","last_undegraded":"2024-04-22T08:28:33.656479+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21049344,"num_objects":5139,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2191},{"shards":"0,2","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5126","reported_seq":5267,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:33.588226+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:33.588226+0000","last_peered":"2024-04-22T08:28:33.588226+0000","last_clean":"2024-04-22T08:28:33.588226+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:33.588226+0000","last_undegraded":"2024-04-22T08:28:33.588226+0000","last_fullsized":"2024-04-22T08:28:33.588226+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20987904,"num_objects":5124,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":7096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5383","reported_seq":5503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.484791+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:34.484791+0000","last_peered":"2024-04-22T08:28:34.484791+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:34.484791+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21783762,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5323,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5199","reported_seq":5270,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.678338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:33.678338+0000","last_peered":"2024-04-22T08:28:33.678338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:33.678338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21263570,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":20768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1422},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5209","reported_seq":5352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.461097+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:34.461097+0000","last_peered":"2024-04-22T08:28:34.461097+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:34.461097+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21315584,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1663,"num_write_kb":6652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5264","reported_seq":5316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:34.503971+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:34.503971+0000","last_peered":"2024-04-22T08:28:34.503971+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:34.503971+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21538002,"num_objects":5260,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":21035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1457}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5111","reported_seq":5187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.595848+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:33.595848+0000","last_peered":"2024-04-22T08:28:33.595848+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:33.595848+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20927081,"num_objects":5110,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1389},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5230","reported_seq":5402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.631996+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:33.631996+0000","last_peered":"2024-04-22T08:28:33.631996+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:33.631996+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21389312,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":5844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1385},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5171","reported_seq":5244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.639110+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:33.639110+0000","last_peered":"2024-04-22T08:28:33.639110+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:33.639110+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21165266,"num_objects":5169,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":20670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1464},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5296","reported_seq":5437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:33.653233+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:33.653233+0000","last_peered":"2024-04-22T08:28:33.653233+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:33.653233+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21645732,"num_objects":5288,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5293,"num_write_kb":21143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1407},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":341847893,"num_objects":83470,"num_object_clones":0,"num_object_copies":166940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2948,"num_objects_unfound":0,"num_objects_dirty":83470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57072,"num_write_kb":228201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":3428969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693227520,"data_stored":693140545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83597,"ondisk_log_size":83597,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117015,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450872,"kb_used_data":288816,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93920968,"statfs":{"total":96636764160,"available":96175071232,"internally_reserved":0,"allocated":295747584,"data_stored":295546569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165932751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":3.9780000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6029999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920778,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185420,"kb_used_data":77508,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94186420,"statfs":{"total":96636764160,"available":96446894080,"internally_reserved":0,"allocated":79368192,"data_stored":79090639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":110489292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313242,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486380,"kb_used_data":310436,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93885460,"statfs":{"total":96636764160,"available":96138711040,"internally_reserved":0,"allocated":317886464,"data_stored":317691911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317939712,"data_stored":317936233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79192064,"data_stored":79112047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296095744,"data_stored":296092265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'5227", 'reported_seq': 5287, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.657759+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:33.657759+0000', 'last_peered': '2024-04-22T08:28:33.657759+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:33.657759+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5227, 'log_dups_size': 0, 'ondisk_log_size': 5227, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21409792, 'num_objects': 5227, 'num_object_clones': 0, 'num_object_copies': 10454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5227, 'num_write_kb': 20908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1433}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'5224", 'reported_seq': 5343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:34.529065+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:34.529065+0000', 'last_peered': '2024-04-22T08:28:34.529065+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:34.529065+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 5224, 'log_dups_size': 0, 'ondisk_log_size': 5224, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21381120, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 10440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2051, 'num_write_kb': 8204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 1465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'5280", 'reported_seq': 5436, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.565951+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:33.565951+0000', 'last_peered': '2024-04-22T08:28:33.565951+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:33.565951+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 5280, 'log_dups_size': 0, 'ondisk_log_size': 5280, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21606400, 'num_objects': 5275, 'num_object_clones': 0, 'num_object_copies': 10550, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2149, 'num_write_kb': 8596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1396}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'5145", 'reported_seq': 6970, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:33.656479+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:33.656479+0000', 'last_peered': '2024-04-22T08:28:33.656479+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:33.656479+0000', 'last_undegraded': '2024-04-22T08:28:33.656479+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 5145, 'log_dups_size': 0, 'ondisk_log_size': 5145, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21049344, 'num_objects': 5139, 'num_object_clones': 0, 'num_object_copies': 10278, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2948, 'num_objects_unfound': 0, 'num_objects_dirty': 5139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1513, 'num_write_kb': 6052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 833, 'num_bytes_recovered': 3411968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2191}, {'shards': '0,2', 'objects': 2948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'5383", 'reported_seq': 5503, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:34.484791+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:34.484791+0000', 'last_peered': '2024-04-22T08:28:34.484791+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:34.484791+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 5383, 'log_dups_size': 0, 'ondisk_log_size': 5383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21783762, 'num_objects': 5320, 'num_object_clones': 0, 'num_object_copies': 10640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5323, 'num_write_kb': 21276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 1432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'5199", 'reported_seq': 5270, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.678338+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:33.678338+0000', 'last_peered': '2024-04-22T08:28:33.678338+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:33.678338+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5199, 'log_dups_size': 0, 'ondisk_log_size': 5199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21263570, 'num_objects': 5193, 'num_object_clones': 0, 'num_object_copies': 10386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5197, 'num_write_kb': 20768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1422}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'5209", 'reported_seq': 5352, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:34.461097+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:34.461097+0000', 'last_peered': '2024-04-22T08:28:34.461097+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:34.461097+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 5209, 'log_dups_size': 0, 'ondisk_log_size': 5209, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21315584, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 10408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1663, 'num_write_kb': 6652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 1454}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'5264", 'reported_seq': 5316, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:34.503971+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:34.503971+0000', 'last_peered': '2024-04-22T08:28:34.503971+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:34.503971+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5264, 'log_dups_size': 0, 'ondisk_log_size': 5264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21538002, 'num_objects': 5260, 'num_object_clones': 0, 'num_object_copies': 10520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5262, 'num_write_kb': 21035, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 1457}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'5111", 'reported_seq': 5187, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.595848+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:33.595848+0000', 'last_peered': '2024-04-22T08:28:33.595848+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:33.595848+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5111, 'log_dups_size': 0, 'ondisk_log_size': 5111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20927081, 'num_objects': 5110, 'num_object_clones': 0, 'num_object_copies': 10220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5110, 'num_write_kb': 20437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1389}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:39.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'5230", 'reported_seq': 5402, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.631996+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:33.631996+0000', 'last_peered': '2024-04-22T08:28:33.631996+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:33.631996+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 5230, 'log_dups_size': 0, 'ondisk_log_size': 5230, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21389312, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 10444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1461, 'num_write_kb': 5844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1385}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:39.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'5171", 'reported_seq': 5244, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.639110+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:33.639110+0000', 'last_peered': '2024-04-22T08:28:33.639110+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:33.639110+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5171, 'log_dups_size': 0, 'ondisk_log_size': 5171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21165266, 'num_objects': 5169, 'num_object_clones': 0, 'num_object_copies': 10338, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5169, 'num_write_kb': 20670, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1464}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:39.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'5296", 'reported_seq': 5437, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:33.653233+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:33.653233+0000', 'last_peered': '2024-04-22T08:28:33.653233+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:33.653233+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 5296, 'log_dups_size': 0, 'ondisk_log_size': 5296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21645732, 'num_objects': 5288, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5293, 'num_write_kb': 21143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1407}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:39.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:39.833 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 87 20 86669 86649 3.88952 1.89844 0.00490811 0.0200462 2024-04-22T08:28:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:40.083 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/166940 objects degraded (25.041%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":83470,"data_bytes":341847893,"bytes_used":1149616128,"bytes_avail":288760676352,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":166940,"degraded_ratio":0.25041332215167128,"misplaced_objects":2948,"misplaced_total":166940,"misplaced_ratio":0.017659039175751769,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":86331,"recovering_keys_per_sec":0,"num_objects_recovered":253,"num_bytes_recovered":1036288,"num_keys_recovered":0,"write_bytes_sec":1988007,"read_op_per_sec":0,"write_op_per_sec":485},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-22T08:27:50.148392+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:40.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:40.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:40.838 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 88 20 87192 87172 3.86835 2.04297 0.00438815 0.0201837 2024-04-22T08:28:41.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:41.005 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:41.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-04-22T08:28:40.166056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353566549,"num_objects":86331,"num_object_clones":0,"num_object_copies":172662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2829,"num_objects_unfound":0,"num_objects_dirty":86331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59933,"num_write_kb":239645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":3916393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86458,"ondisk_log_size":86458,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1149064,"kb_used_data":690864,"kb_used_omap":12,"kb_used_meta":458163,"kb_avail":281966456,"statfs":{"total":289910292480,"available":288733650944,"internally_reserved":0,"allocated":707444736,"data_stored":706771615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":469159022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,6,4,2,20],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35581952,"num_objects":8687,"num_object_clones":0,"num_object_copies":17374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-372,"num_objects_unfound":0,"num_objects_dirty":8687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8687,"num_write_kb":34748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003501"},"pg_stats":[{"pgid":"2.f","version":"35'5424","reported_seq":5483,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.762727+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:38.762727+0000","last_peered":"2024-04-22T08:28:38.762727+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:38.762727+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":5424,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1630},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5422","reported_seq":5534,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.549223+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:39.549223+0000","last_peered":"2024-04-22T08:28:39.549223+0000","last_clean":"2024-04-22T08:28:39.549223+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:39.549223+0000","last_undegraded":"2024-04-22T08:28:39.549223+0000","last_fullsized":"2024-04-22T08:28:39.549223+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22183936,"num_objects":5416,"num_object_clones":0,"num_object_copies":10832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5407","reported_seq":5527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.539468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:39.539468+0000","last_peered":"2024-04-22T08:28:39.539468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:39.539468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":8936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5467","reported_seq":5494,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.467559+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:39.467559+0000","last_peered":"2024-04-22T08:28:39.467559+0000","last_clean":"2024-04-22T08:28:39.467559+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:39.467559+0000","last_undegraded":"2024-04-22T08:28:39.467559+0000","last_fullsized":"2024-04-22T08:28:39.467559+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5377","reported_seq":5512,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:38.773465+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:38.773465+0000","last_peered":"2024-04-22T08:28:38.773465+0000","last_clean":"2024-04-22T08:28:38.773465+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:38.773465+0000","last_undegraded":"2024-04-22T08:28:38.773465+0000","last_fullsized":"2024-04-22T08:28:38.773465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5377,"log_dups_size":0,"ondisk_log_size":5377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21995520,"num_objects":5370,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5478","reported_seq":5637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.700894+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:38.700894+0000","last_peered":"2024-04-22T08:28:38.700894+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:38.700894+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22417408,"num_objects":5473,"num_object_clones":0,"num_object_copies":10946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":9388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1594},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5318","reported_seq":7380,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:38.755728+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:38.755728+0000","last_peered":"2024-04-22T08:28:38.755728+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:38.755728+0000","last_undegraded":"2024-04-22T08:28:38.755728+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":5312,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2829,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1686,"num_write_kb":6744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":3899392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2483},{"shards":"0,2","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5316","reported_seq":5456,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:38.747438+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:38.747438+0000","last_peered":"2024-04-22T08:28:38.747438+0000","last_clean":"2024-04-22T08:28:38.747438+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:38.747438+0000","last_undegraded":"2024-04-22T08:28:38.747438+0000","last_fullsized":"2024-04-22T08:28:38.747438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21766144,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1964,"num_write_kb":7856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5555","reported_seq":5675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.515554+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:39.515554+0000","last_peered":"2024-04-22T08:28:39.515554+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:39.515554+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488274,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5378","reported_seq":5452,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.703138+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:38.703138+0000","last_peered":"2024-04-22T08:28:38.703138+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:38.703138+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21996754,"num_objects":5372,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":21484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1601},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5403","reported_seq":5545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.528565+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:39.528565+0000","last_peered":"2024-04-22T08:28:39.528565+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:39.528565+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22110208,"num_objects":5398,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1857,"num_write_kb":7428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5419","reported_seq":5470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.554005+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:39.554005+0000","last_peered":"2024-04-22T08:28:39.554005+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:39.554005+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172882,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":21655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5315","reported_seq":5391,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.654790+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:38.654790+0000","last_peered":"2024-04-22T08:28:38.654790+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:38.654790+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5315,"log_dups_size":0,"ondisk_log_size":5315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21762665,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":21253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1593},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5392","reported_seq":5564,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.702712+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:38.702712+0000","last_peered":"2024-04-22T08:28:38.702712+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:38.702712+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22052864,"num_objects":5384,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1623,"num_write_kb":6492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1547},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5346","reported_seq":5419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.669337+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:38.669337+0000","last_peered":"2024-04-22T08:28:38.669337+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:38.669337+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5346,"log_dups_size":0,"ondisk_log_size":5346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21882066,"num_objects":5344,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":21370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1639},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5441","reported_seq":5582,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:38.744563+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:38.744563+0000","last_peered":"2024-04-22T08:28:38.744563+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:38.744563+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22239652,"num_objects":5433,"num_object_clones":0,"num_object_copies":10866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5438,"num_write_kb":21723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1552},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":353566549,"num_objects":86331,"num_object_clones":0,"num_object_copies":172662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2829,"num_objects_unfound":0,"num_objects_dirty":86331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59933,"num_write_kb":239645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":3916393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":707817472,"data_stored":707730497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86458,"ondisk_log_size":86458,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117016,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460640,"kb_used_data":294488,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93911200,"statfs":{"total":96636764160,"available":96165068800,"internally_reserved":0,"allocated":301555712,"data_stored":301354697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170127055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.221},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920778,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185420,"kb_used_data":77508,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94186420,"statfs":{"total":96636764160,"available":96446894080,"internally_reserved":0,"allocated":79368192,"data_stored":79090639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":110489292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313243,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":503004,"kb_used_data":318868,"kb_used_omap":4,"kb_used_meta":184123,"kb_avail":93868836,"statfs":{"total":96636764160,"available":96121688064,"internally_reserved":0,"allocated":326520832,"data_stored":326326279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188542675},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,20],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326758400,"data_stored":326754921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79192064,"data_stored":79112047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301867008,"data_stored":301863529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:41.838 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 89 20 87727 87707 3.84837 2.08984 0.0690769 0.0202924 2024-04-22T08:28:42.838 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 90 20 88298 88278 3.83039 2.23047 0.0362002 0.0203887 2024-04-22T08:28:43.838 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 91 20 88870 88850 3.81286 2.23438 0.0472428 0.0204826 2024-04-22T08:28:44.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:44.841 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 92 20 89415 89395 3.79448 2.12891 0.0702371 0.0205798 2024-04-22T08:28:44.920 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:44.920 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:44.979 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-22T08:28:44.166975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":360427349,"num_objects":88006,"num_object_clones":0,"num_object_copies":176012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":88006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61608,"num_write_kb":246345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88133,"ondisk_log_size":88133,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1166848,"kb_used_data":706984,"kb_used_omap":12,"kb_used_meta":459827,"kb_avail":281948672,"statfs":{"total":289910292480,"available":288715440128,"internally_reserved":0,"allocated":723951616,"data_stored":723278495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":470862958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,10,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30400512,"num_objects":7422,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-362,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7422,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002824"},"pg_stats":[{"pgid":"2.f","version":"35'5590","reported_seq":5649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.810982+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:43.810982+0000","last_peered":"2024-04-22T08:28:43.810982+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:43.810982+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22896640,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":22360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1796},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5422","reported_seq":5534,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.549223+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:39.549223+0000","last_peered":"2024-04-22T08:28:39.549223+0000","last_clean":"2024-04-22T08:28:39.549223+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:39.549223+0000","last_undegraded":"2024-04-22T08:28:39.549223+0000","last_fullsized":"2024-04-22T08:28:39.549223+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22183936,"num_objects":5416,"num_object_clones":0,"num_object_copies":10832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5407","reported_seq":5527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.539468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:39.539468+0000","last_peered":"2024-04-22T08:28:39.539468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:39.539468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":8936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5467","reported_seq":5494,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.467559+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:39.467559+0000","last_peered":"2024-04-22T08:28:39.467559+0000","last_clean":"2024-04-22T08:28:39.467559+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:39.467559+0000","last_undegraded":"2024-04-22T08:28:39.467559+0000","last_fullsized":"2024-04-22T08:28:39.467559+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5538","reported_seq":5673,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.746580+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:43.746580+0000","last_peered":"2024-04-22T08:28:43.746580+0000","last_clean":"2024-04-22T08:28:43.746580+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:43.746580+0000","last_undegraded":"2024-04-22T08:28:43.746580+0000","last_fullsized":"2024-04-22T08:28:43.746580+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22654976,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":10520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5647","reported_seq":5804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.785609+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:43.785609+0000","last_peered":"2024-04-22T08:28:43.785609+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:43.785609+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23109632,"num_objects":5642,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1763},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5478","reported_seq":7776,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:43.793857+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:43.793857+0000","last_peered":"2024-04-22T08:28:43.793857+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:43.793857+0000","last_undegraded":"2024-04-22T08:28:43.793857+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":5472,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":7384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":4386816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5479","reported_seq":5620,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.790194+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:43.790194+0000","last_peered":"2024-04-22T08:28:43.790194+0000","last_clean":"2024-04-22T08:28:43.790194+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:43.790194+0000","last_undegraded":"2024-04-22T08:28:43.790194+0000","last_fullsized":"2024-04-22T08:28:43.790194+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22433792,"num_objects":5477,"num_object_clones":0,"num_object_copies":10954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5555","reported_seq":5675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.515554+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:39.515554+0000","last_peered":"2024-04-22T08:28:39.515554+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:39.515554+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488274,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5570","reported_seq":5644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.684246+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:43.684246+0000","last_peered":"2024-04-22T08:28:43.684246+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:43.684246+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22783186,"num_objects":5564,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":22252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5403","reported_seq":5545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.528565+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:39.528565+0000","last_peered":"2024-04-22T08:28:39.528565+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:39.528565+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22110208,"num_objects":5398,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1857,"num_write_kb":7428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5419","reported_seq":5470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.554005+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:39.554005+0000","last_peered":"2024-04-22T08:28:39.554005+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:39.554005+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172882,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":21655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5498","reported_seq":5574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.778068+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:43.778068+0000","last_peered":"2024-04-22T08:28:43.778068+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:43.778068+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22512233,"num_objects":5497,"num_object_clones":0,"num_object_copies":10994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":21985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1776},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5538","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.719011+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:43.719011+0000","last_peered":"2024-04-22T08:28:43.719011+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:43.719011+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22650880,"num_objects":5530,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1769,"num_write_kb":7076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1693},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5524","reported_seq":5597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.769061+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:43.769061+0000","last_peered":"2024-04-22T08:28:43.769061+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:43.769061+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22611154,"num_objects":5522,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":22082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1817},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5598","reported_seq":5739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.727537+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:43.727537+0000","last_peered":"2024-04-22T08:28:43.727537+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:43.727537+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22882724,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":22351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1709},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":360427349,"num_objects":88006,"num_object_clones":0,"num_object_copies":176012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":88006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61608,"num_write_kb":246345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724410368,"data_stored":724323393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88133,"ondisk_log_size":88133,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117016,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460640,"kb_used_data":294488,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93911200,"statfs":{"total":96636764160,"available":96165068800,"internally_reserved":0,"allocated":301555712,"data_stored":301354697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170127055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.221},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":203332,"kb_used_data":85564,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94168508,"statfs":{"total":96636764160,"available":96428552192,"internally_reserved":0,"allocated":87617536,"data_stored":87339983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":120581836},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7480000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313244,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502876,"kb_used_data":326932,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93868964,"statfs":{"total":96636764160,"available":96121819136,"internally_reserved":0,"allocated":334778368,"data_stored":334583815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.79400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335028224,"data_stored":335024745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87435119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301867008,"data_stored":301863529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:44.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:45.793 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:45.798 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:45.840 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 93 20 89956 89936 3.7764 2.11328 0.0390857 0.0206801 2024-04-22T08:28:45.846 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-22T08:28:44.166975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":360427349,"num_objects":88006,"num_object_clones":0,"num_object_copies":176012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":88006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61608,"num_write_kb":246345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88133,"ondisk_log_size":88133,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1166848,"kb_used_data":706984,"kb_used_omap":12,"kb_used_meta":459827,"kb_avail":281948672,"statfs":{"total":289910292480,"available":288715440128,"internally_reserved":0,"allocated":723951616,"data_stored":723278495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":470862958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,10,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30400512,"num_objects":7422,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-362,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7422,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002824"},"pg_stats":[{"pgid":"2.f","version":"35'5590","reported_seq":5649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.810982+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:43.810982+0000","last_peered":"2024-04-22T08:28:43.810982+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:43.810982+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22896640,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":22360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1796},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5422","reported_seq":5534,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.549223+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:39.549223+0000","last_peered":"2024-04-22T08:28:39.549223+0000","last_clean":"2024-04-22T08:28:39.549223+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:39.549223+0000","last_undegraded":"2024-04-22T08:28:39.549223+0000","last_fullsized":"2024-04-22T08:28:39.549223+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22183936,"num_objects":5416,"num_object_clones":0,"num_object_copies":10832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5407","reported_seq":5527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.539468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:39.539468+0000","last_peered":"2024-04-22T08:28:39.539468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:39.539468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":8936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5467","reported_seq":5494,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:39.467559+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:39.467559+0000","last_peered":"2024-04-22T08:28:39.467559+0000","last_clean":"2024-04-22T08:28:39.467559+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:39.467559+0000","last_undegraded":"2024-04-22T08:28:39.467559+0000","last_fullsized":"2024-04-22T08:28:39.467559+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5538","reported_seq":5673,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.746580+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:43.746580+0000","last_peered":"2024-04-22T08:28:43.746580+0000","last_clean":"2024-04-22T08:28:43.746580+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:43.746580+0000","last_undegraded":"2024-04-22T08:28:43.746580+0000","last_fullsized":"2024-04-22T08:28:43.746580+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22654976,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":10520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5647","reported_seq":5804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.785609+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:43.785609+0000","last_peered":"2024-04-22T08:28:43.785609+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:43.785609+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23109632,"num_objects":5642,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1763},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5478","reported_seq":7776,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:43.793857+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:43.793857+0000","last_peered":"2024-04-22T08:28:43.793857+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:43.793857+0000","last_undegraded":"2024-04-22T08:28:43.793857+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":5472,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":7384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":4386816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5479","reported_seq":5620,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.790194+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:43.790194+0000","last_peered":"2024-04-22T08:28:43.790194+0000","last_clean":"2024-04-22T08:28:43.790194+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:43.790194+0000","last_undegraded":"2024-04-22T08:28:43.790194+0000","last_fullsized":"2024-04-22T08:28:43.790194+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22433792,"num_objects":5477,"num_object_clones":0,"num_object_copies":10954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5555","reported_seq":5675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.515554+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:39.515554+0000","last_peered":"2024-04-22T08:28:39.515554+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:39.515554+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488274,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5570","reported_seq":5644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.684246+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:43.684246+0000","last_peered":"2024-04-22T08:28:43.684246+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:43.684246+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22783186,"num_objects":5564,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":22252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5403","reported_seq":5545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.528565+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:39.528565+0000","last_peered":"2024-04-22T08:28:39.528565+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:39.528565+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22110208,"num_objects":5398,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1857,"num_write_kb":7428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5419","reported_seq":5470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:39.554005+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:39.554005+0000","last_peered":"2024-04-22T08:28:39.554005+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:39.554005+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172882,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":21655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5498","reported_seq":5574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.778068+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:43.778068+0000","last_peered":"2024-04-22T08:28:43.778068+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:43.778068+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22512233,"num_objects":5497,"num_object_clones":0,"num_object_copies":10994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":21985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1776},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5538","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.719011+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:43.719011+0000","last_peered":"2024-04-22T08:28:43.719011+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:43.719011+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22650880,"num_objects":5530,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1769,"num_write_kb":7076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1693},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5524","reported_seq":5597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.769061+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:43.769061+0000","last_peered":"2024-04-22T08:28:43.769061+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:43.769061+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22611154,"num_objects":5522,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":22082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1817},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5598","reported_seq":5739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.727537+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:43.727537+0000","last_peered":"2024-04-22T08:28:43.727537+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:43.727537+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22882724,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":22351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1709},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":360427349,"num_objects":88006,"num_object_clones":0,"num_object_copies":176012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":88006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61608,"num_write_kb":246345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":724410368,"data_stored":724323393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88133,"ondisk_log_size":88133,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117016,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460640,"kb_used_data":294488,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93911200,"statfs":{"total":96636764160,"available":96165068800,"internally_reserved":0,"allocated":301555712,"data_stored":301354697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170127055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.221},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":203332,"kb_used_data":85564,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94168508,"statfs":{"total":96636764160,"available":96428552192,"internally_reserved":0,"allocated":87617536,"data_stored":87339983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":120581836},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7480000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313244,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502876,"kb_used_data":326932,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93868964,"statfs":{"total":96636764160,"available":96121819136,"internally_reserved":0,"allocated":334778368,"data_stored":334583815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.79400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335028224,"data_stored":335024745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87435119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301867008,"data_stored":301863529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'5590", 'reported_seq': 5649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.810982+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:43.810982+0000', 'last_peered': '2024-04-22T08:28:43.810982+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:43.810982+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5590, 'log_dups_size': 0, 'ondisk_log_size': 5590, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22896640, 'num_objects': 5590, 'num_object_clones': 0, 'num_object_copies': 11180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5590, 'num_write_kb': 22360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1796}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'5407", 'reported_seq': 5527, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:39.539468+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:39.539468+0000', 'last_peered': '2024-04-22T08:28:39.539468+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:39.539468+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 5407, 'log_dups_size': 0, 'ondisk_log_size': 5407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22130688, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 10806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2234, 'num_write_kb': 8936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 1648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:45.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'5647", 'reported_seq': 5804, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.785609+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:43.785609+0000', 'last_peered': '2024-04-22T08:28:43.785609+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:43.785609+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 5647, 'log_dups_size': 0, 'ondisk_log_size': 5647, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23109632, 'num_objects': 5642, 'num_object_clones': 0, 'num_object_copies': 11284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2516, 'num_write_kb': 10064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1763}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'5478", 'reported_seq': 7776, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:43.793857+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:43.793857+0000', 'last_peered': '2024-04-22T08:28:43.793857+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:43.793857+0000', 'last_undegraded': '2024-04-22T08:28:43.793857+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 5478, 'log_dups_size': 0, 'ondisk_log_size': 5478, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22413312, 'num_objects': 5472, 'num_object_clones': 0, 'num_object_copies': 10944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2710, 'num_objects_unfound': 0, 'num_objects_dirty': 5472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1846, 'num_write_kb': 7384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1071, 'num_bytes_recovered': 4386816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2762}, {'shards': '0,2', 'objects': 2710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'5555", 'reported_seq': 5675, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:39.515554+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:39.515554+0000', 'last_peered': '2024-04-22T08:28:39.515554+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:39.515554+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 5555, 'log_dups_size': 0, 'ondisk_log_size': 5555, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22488274, 'num_objects': 5492, 'num_object_clones': 0, 'num_object_copies': 10984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5495, 'num_write_kb': 21964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 1604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'5570", 'reported_seq': 5644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.684246+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:43.684246+0000', 'last_peered': '2024-04-22T08:28:43.684246+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:43.684246+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5570, 'log_dups_size': 0, 'ondisk_log_size': 5570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22783186, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 11128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5568, 'num_write_kb': 22252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1793}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'5403", 'reported_seq': 5545, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:39.528565+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:39.528565+0000', 'last_peered': '2024-04-22T08:28:39.528565+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:39.528565+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 5403, 'log_dups_size': 0, 'ondisk_log_size': 5403, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22110208, 'num_objects': 5398, 'num_object_clones': 0, 'num_object_copies': 10796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1857, 'num_write_kb': 7428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 1648}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:45.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'5419", 'reported_seq': 5470, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:39.554005+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:39.554005+0000', 'last_peered': '2024-04-22T08:28:39.554005+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:39.554005+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5419, 'log_dups_size': 0, 'ondisk_log_size': 5419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22172882, 'num_objects': 5415, 'num_object_clones': 0, 'num_object_copies': 10830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5417, 'num_write_kb': 21655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 1612}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'5498", 'reported_seq': 5574, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.778068+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:43.778068+0000', 'last_peered': '2024-04-22T08:28:43.778068+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:43.778068+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5498, 'log_dups_size': 0, 'ondisk_log_size': 5498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22512233, 'num_objects': 5497, 'num_object_clones': 0, 'num_object_copies': 10994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5497, 'num_write_kb': 21985, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1776}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'5538", 'reported_seq': 5710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.719011+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:43.719011+0000', 'last_peered': '2024-04-22T08:28:43.719011+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:43.719011+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 5538, 'log_dups_size': 0, 'ondisk_log_size': 5538, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22650880, 'num_objects': 5530, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1769, 'num_write_kb': 7076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1693}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'5524", 'reported_seq': 5597, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.769061+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:43.769061+0000', 'last_peered': '2024-04-22T08:28:43.769061+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:43.769061+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5524, 'log_dups_size': 0, 'ondisk_log_size': 5524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22611154, 'num_objects': 5522, 'num_object_clones': 0, 'num_object_copies': 11044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5522, 'num_write_kb': 22082, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1817}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:45.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'5598", 'reported_seq': 5739, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:43.727537+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:43.727537+0000', 'last_peered': '2024-04-22T08:28:43.727537+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:43.727537+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 5598, 'log_dups_size': 0, 'ondisk_log_size': 5598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22882724, 'num_objects': 5590, 'num_object_clones': 0, 'num_object_copies': 11180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5595, 'num_write_kb': 22351, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1709}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:45.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:46.742 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/178032 objects degraded (23.481%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":89016,"data_bytes":364564309,"bytes_used":1187733504,"bytes_avail":288722558976,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":178032,"degraded_ratio":0.23481171924148467,"misplaced_objects":2710,"misplaced_total":178032,"misplaced_ratio":0.015221982564932146,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":81218,"recovering_keys_per_sec":0,"num_objects_recovered":238,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":2251246,"read_op_per_sec":0,"write_op_per_sec":549},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-22T08:28:46.167488+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:28:46.809 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:46.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:46.840 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 94 19 90481 90462 3.75809 2.05469 0.059933 0.0207793 2024-04-22T08:28:47.685 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:47.685 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:47.717 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-22T08:28:46.167433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364564309,"num_objects":89016,"num_object_clones":0,"num_object_copies":178032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":89016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62618,"num_write_kb":250385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89143,"ondisk_log_size":89143,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1159896,"kb_used_data":712320,"kb_used_omap":12,"kb_used_meta":447539,"kb_avail":281955624,"statfs":{"total":289910292480,"available":288722558976,"internally_reserved":0,"allocated":729415680,"data_stored":728742559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":458280046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,10,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27021312,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-238,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":26388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"2.f","version":"35'5590","reported_seq":5649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.810982+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:43.810982+0000","last_peered":"2024-04-22T08:28:43.810982+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:43.810982+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22896640,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":22360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1796},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5617","reported_seq":5729,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:44.594441+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:44.594441+0000","last_peered":"2024-04-22T08:28:44.594441+0000","last_clean":"2024-04-22T08:28:44.594441+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:44.594441+0000","last_undegraded":"2024-04-22T08:28:44.594441+0000","last_fullsized":"2024-04-22T08:28:44.594441+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22982656,"num_objects":5611,"num_object_clones":0,"num_object_copies":11222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":10692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5555","reported_seq":5674,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:44.540751+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:44.540751+0000","last_peered":"2024-04-22T08:28:44.540751+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:44.540751+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22736896,"num_objects":5551,"num_object_clones":0,"num_object_copies":11102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":9528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1796}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5636","reported_seq":5663,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:44.563484+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:44.563484+0000","last_peered":"2024-04-22T08:28:44.563484+0000","last_clean":"2024-04-22T08:28:44.563484+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:44.563484+0000","last_undegraded":"2024-04-22T08:28:44.563484+0000","last_fullsized":"2024-04-22T08:28:44.563484+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5636,"log_dups_size":0,"ondisk_log_size":5636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085056,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":22544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5538","reported_seq":5673,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.746580+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:43.746580+0000","last_peered":"2024-04-22T08:28:43.746580+0000","last_clean":"2024-04-22T08:28:43.746580+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:43.746580+0000","last_undegraded":"2024-04-22T08:28:43.746580+0000","last_fullsized":"2024-04-22T08:28:43.746580+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22654976,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":10520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5647","reported_seq":5804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.785609+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:43.785609+0000","last_peered":"2024-04-22T08:28:43.785609+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:43.785609+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23109632,"num_objects":5642,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1763},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5478","reported_seq":7776,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:43.793857+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:43.793857+0000","last_peered":"2024-04-22T08:28:43.793857+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:43.793857+0000","last_undegraded":"2024-04-22T08:28:43.793857+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":5472,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":7384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":4386816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5479","reported_seq":5620,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:43.790194+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:43.790194+0000","last_peered":"2024-04-22T08:28:43.790194+0000","last_clean":"2024-04-22T08:28:43.790194+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:43.790194+0000","last_undegraded":"2024-04-22T08:28:43.790194+0000","last_fullsized":"2024-04-22T08:28:43.790194+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22433792,"num_objects":5477,"num_object_clones":0,"num_object_copies":10954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5706","reported_seq":5826,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:44.440954+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:44.440954+0000","last_peered":"2024-04-22T08:28:44.440954+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:44.440954+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23106770,"num_objects":5643,"num_object_clones":0,"num_object_copies":11286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":22568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5570","reported_seq":5644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.684246+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:43.684246+0000","last_peered":"2024-04-22T08:28:43.684246+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:43.684246+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22783186,"num_objects":5564,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":22252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5573","reported_seq":5712,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:44.546576+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:44.546576+0000","last_peered":"2024-04-22T08:28:44.546576+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:44.546576+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5573,"log_dups_size":0,"ondisk_log_size":5573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":5568,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":8108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5596","reported_seq":5648,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:44.515754+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:44.515754+0000","last_peered":"2024-04-22T08:28:44.515754+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:44.515754+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22897874,"num_objects":5592,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":22363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5498","reported_seq":5574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.778068+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:43.778068+0000","last_peered":"2024-04-22T08:28:43.778068+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:43.778068+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22512233,"num_objects":5497,"num_object_clones":0,"num_object_copies":10994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":21985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1776},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5538","reported_seq":5710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.719011+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:43.719011+0000","last_peered":"2024-04-22T08:28:43.719011+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:43.719011+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22650880,"num_objects":5530,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1769,"num_write_kb":7076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1693},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5524","reported_seq":5597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.769061+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:43.769061+0000","last_peered":"2024-04-22T08:28:43.769061+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:43.769061+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22611154,"num_objects":5522,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":22082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1817},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5598","reported_seq":5739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:43.727537+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:43.727537+0000","last_peered":"2024-04-22T08:28:43.727537+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:43.727537+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22882724,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":22351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1709},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":364564309,"num_objects":89016,"num_object_clones":0,"num_object_copies":178032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2710,"num_objects_unfound":0,"num_objects_dirty":89016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62618,"num_write_kb":250385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":4403817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729878528,"data_stored":729791553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89143,"ondisk_log_size":89143,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117017,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":453688,"kb_used_data":299824,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":93918152,"statfs":{"total":96636764160,"available":96172187648,"internally_reserved":0,"allocated":307019776,"data_stored":306818761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157544143},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":203332,"kb_used_data":85564,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94168508,"statfs":{"total":96636764160,"available":96428552192,"internally_reserved":0,"allocated":87617536,"data_stored":87339983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":120581836},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7480000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313244,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502876,"kb_used_data":326932,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93868964,"statfs":{"total":96636764160,"available":96121819136,"internally_reserved":0,"allocated":334778368,"data_stored":334583815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":180154067},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.79400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335028224,"data_stored":335024745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87515136,"data_stored":87435119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307335168,"data_stored":307331689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:47.840 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 95 20 91026 91006 3.7409 2.125 0.0275967 0.0208764 2024-04-22T08:28:48.840 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 96 20 91600 91580 3.72529 2.24219 0.029849 0.0209615 2024-04-22T08:28:49.841 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 97 20 92164 92144 3.70957 2.20312 0.0254287 0.0210522 2024-04-22T08:28:50.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:50.841 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 98 20 92726 92706 3.69412 2.19531 0.0324728 0.0211382 2024-04-22T08:28:51.658 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:51.659 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:51.721 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-22T08:28:50.168174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":375824213,"num_objects":91765,"num_object_clones":0,"num_object_copies":183530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":91765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65367,"num_write_kb":261381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":4911721,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91892,"ondisk_log_size":91892,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1216800,"kb_used_data":734792,"kb_used_omap":12,"kb_used_meta":481971,"kb_avail":281898720,"statfs":{"total":289910292480,"available":288664289280,"internally_reserved":0,"allocated":752427008,"data_stored":751753887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":493538414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,13,4,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33976320,"num_objects":8295,"num_object_clones":0,"num_object_copies":16590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-362,"num_objects_unfound":0,"num_objects_dirty":8295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":33180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002561"},"pg_stats":[{"pgid":"2.f","version":"35'5774","reported_seq":5834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.845144+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:48.845144+0000","last_peered":"2024-04-22T08:28:48.845144+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:48.845144+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":23096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5779","reported_seq":5891,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.521811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:49.521811+0000","last_peered":"2024-04-22T08:28:49.521811+0000","last_clean":"2024-04-22T08:28:49.521811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:49.521811+0000","last_undegraded":"2024-04-22T08:28:49.521811+0000","last_fullsized":"2024-04-22T08:28:49.521811+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23646208,"num_objects":5773,"num_object_clones":0,"num_object_copies":11546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2835,"num_write_kb":11340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5708","reported_seq":5828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.577668+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:49.577668+0000","last_peered":"2024-04-22T08:28:49.577668+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:49.577668+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":5704,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":10140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5799","reported_seq":5826,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.590595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:49.590595+0000","last_peered":"2024-04-22T08:28:49.590595+0000","last_clean":"2024-04-22T08:28:49.590595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:49.590595+0000","last_undegraded":"2024-04-22T08:28:49.590595+0000","last_fullsized":"2024-04-22T08:28:49.590595+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752704,"num_objects":5799,"num_object_clones":0,"num_object_copies":11598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5717","reported_seq":5852,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:48.805005+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:48.805005+0000","last_peered":"2024-04-22T08:28:48.805005+0000","last_clean":"2024-04-22T08:28:48.805005+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:48.805005+0000","last_undegraded":"2024-04-22T08:28:48.805005+0000","last_fullsized":"2024-04-22T08:28:48.805005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23388160,"num_objects":5710,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2809,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5818","reported_seq":5977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.720781+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:48.720781+0000","last_peered":"2024-04-22T08:28:48.720781+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:48.720781+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23810048,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":10748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1934},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5657","reported_seq":8204,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:48.794473+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:48.794473+0000","last_peered":"2024-04-22T08:28:48.794473+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:48.794473+0000","last_undegraded":"2024-04-22T08:28:48.794473+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23146496,"num_objects":5651,"num_object_clones":0,"num_object_copies":11302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":4894720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3065},{"shards":"0,2","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5643","reported_seq":5784,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:48.804399+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:48.804399+0000","last_peered":"2024-04-22T08:28:48.804399+0000","last_clean":"2024-04-22T08:28:48.804399+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:48.804399+0000","last_undegraded":"2024-04-22T08:28:48.804399+0000","last_fullsized":"2024-04-22T08:28:48.804399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5643,"log_dups_size":0,"ondisk_log_size":5643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23105536,"num_objects":5641,"num_object_clones":0,"num_object_copies":11282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5867","reported_seq":5985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.594991+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:49.594991+0000","last_peered":"2024-04-22T08:28:49.594991+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:49.594991+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23766226,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":23212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5746","reported_seq":5820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.834302+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:48.834302+0000","last_peered":"2024-04-22T08:28:48.834302+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:48.834302+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23504082,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1969},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5748","reported_seq":5889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.503867+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:49.503867+0000","last_peered":"2024-04-22T08:28:49.503867+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:49.503867+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23523328,"num_objects":5743,"num_object_clones":0,"num_object_copies":11486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5788","reported_seq":5840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.498242+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:49.498242+0000","last_peered":"2024-04-22T08:28:49.498242+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:49.498242+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23684306,"num_objects":5784,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5654","reported_seq":5730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.795361+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:48.795361+0000","last_peered":"2024-04-22T08:28:48.795361+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:48.795361+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23151209,"num_objects":5653,"num_object_clones":0,"num_object_copies":11306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":22609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1932},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5717","reported_seq":5888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.775625+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:48.775625+0000","last_peered":"2024-04-22T08:28:48.775625+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:48.775625+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23384064,"num_objects":5709,"num_object_clones":0,"num_object_copies":11418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":7792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1872},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5704","reported_seq":5777,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.775616+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:48.775616+0000","last_peered":"2024-04-22T08:28:48.775616+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:48.775616+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23348434,"num_objects":5702,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":22802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1997},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5773","reported_seq":5914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.818410+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:48.818410+0000","last_peered":"2024-04-22T08:28:48.818410+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:48.818410+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23599524,"num_objects":5765,"num_object_clones":0,"num_object_copies":11530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":23051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1884},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":375824213,"num_objects":91765,"num_object_clones":0,"num_object_copies":183530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":91765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65367,"num_write_kb":261381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":4911721,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":752975872,"data_stored":752888897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91892,"ondisk_log_size":91892,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117018,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463100,"kb_used_data":305140,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93908740,"statfs":{"total":96636764160,"available":96162549760,"internally_reserved":0,"allocated":312463360,"data_stored":312262345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920780,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238312,"kb_used_data":94304,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94133528,"statfs":{"total":96636764160,"available":96392732672,"internally_reserved":0,"allocated":96567296,"data_stored":96289743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":147451596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8740000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313245,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515388,"kb_used_data":335348,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93856452,"statfs":{"total":96636764160,"available":96109006848,"internally_reserved":0,"allocated":343396352,"data_stored":343201799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":184348371},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":4.202},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":343564288,"data_stored":343560809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96575488,"data_stored":96495471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312836096,"data_stored":312832617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:51.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:51.841 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 99 20 93275 93255 3.67847 2.14453 0.076891 0.0212265 2024-04-22T08:28:52.290 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:52.293 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:52.356 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-22T08:28:52.169384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":375824213,"num_objects":91765,"num_object_clones":0,"num_object_copies":183530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":91765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65367,"num_write_kb":261381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":4911721,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91892,"ondisk_log_size":91892,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1234012,"kb_used_data":743812,"kb_used_omap":12,"kb_used_meta":490163,"kb_avail":281881508,"statfs":{"total":289910292480,"available":288646664192,"internally_reserved":0,"allocated":761663488,"data_stored":760990367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":501927022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,5,12,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22257664,"num_objects":5434,"num_object_clones":0,"num_object_copies":10868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-243,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":21736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":995328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"2.f","version":"35'5774","reported_seq":5834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.845144+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:48.845144+0000","last_peered":"2024-04-22T08:28:48.845144+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:48.845144+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":23096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5779","reported_seq":5891,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.521811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:49.521811+0000","last_peered":"2024-04-22T08:28:49.521811+0000","last_clean":"2024-04-22T08:28:49.521811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:49.521811+0000","last_undegraded":"2024-04-22T08:28:49.521811+0000","last_fullsized":"2024-04-22T08:28:49.521811+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23646208,"num_objects":5773,"num_object_clones":0,"num_object_copies":11546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2835,"num_write_kb":11340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5708","reported_seq":5828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.577668+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:49.577668+0000","last_peered":"2024-04-22T08:28:49.577668+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:49.577668+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":5704,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":10140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5799","reported_seq":5826,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.590595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:49.590595+0000","last_peered":"2024-04-22T08:28:49.590595+0000","last_clean":"2024-04-22T08:28:49.590595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:49.590595+0000","last_undegraded":"2024-04-22T08:28:49.590595+0000","last_fullsized":"2024-04-22T08:28:49.590595+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752704,"num_objects":5799,"num_object_clones":0,"num_object_copies":11598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5717","reported_seq":5852,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:48.805005+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:48.805005+0000","last_peered":"2024-04-22T08:28:48.805005+0000","last_clean":"2024-04-22T08:28:48.805005+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:48.805005+0000","last_undegraded":"2024-04-22T08:28:48.805005+0000","last_fullsized":"2024-04-22T08:28:48.805005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23388160,"num_objects":5710,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2809,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5818","reported_seq":5977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.720781+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:48.720781+0000","last_peered":"2024-04-22T08:28:48.720781+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:48.720781+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23810048,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":10748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1934},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5657","reported_seq":8204,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:48.794473+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:48.794473+0000","last_peered":"2024-04-22T08:28:48.794473+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:48.794473+0000","last_undegraded":"2024-04-22T08:28:48.794473+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23146496,"num_objects":5651,"num_object_clones":0,"num_object_copies":11302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":4894720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3065},{"shards":"0,2","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5643","reported_seq":5784,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:48.804399+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:48.804399+0000","last_peered":"2024-04-22T08:28:48.804399+0000","last_clean":"2024-04-22T08:28:48.804399+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:48.804399+0000","last_undegraded":"2024-04-22T08:28:48.804399+0000","last_fullsized":"2024-04-22T08:28:48.804399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5643,"log_dups_size":0,"ondisk_log_size":5643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23105536,"num_objects":5641,"num_object_clones":0,"num_object_copies":11282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":9164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5867","reported_seq":5985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.594991+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:49.594991+0000","last_peered":"2024-04-22T08:28:49.594991+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:49.594991+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23766226,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":23212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5746","reported_seq":5820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.834302+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:48.834302+0000","last_peered":"2024-04-22T08:28:48.834302+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:48.834302+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23504082,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1969},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.249956+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.249956+0000","last_peered":"2024-04-22T08:27:56.249956+0000","last_clean":"2024-04-22T08:27:56.249956+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.249956+0000","last_undegraded":"2024-04-22T08:27:56.249956+0000","last_fullsized":"2024-04-22T08:27:56.249956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5748","reported_seq":5889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.503867+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:49.503867+0000","last_peered":"2024-04-22T08:28:49.503867+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:49.503867+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23523328,"num_objects":5743,"num_object_clones":0,"num_object_copies":11486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5788","reported_seq":5840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.498242+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:49.498242+0000","last_peered":"2024-04-22T08:28:49.498242+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:49.498242+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23684306,"num_objects":5784,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5654","reported_seq":5730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.795361+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:48.795361+0000","last_peered":"2024-04-22T08:28:48.795361+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:48.795361+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23151209,"num_objects":5653,"num_object_clones":0,"num_object_copies":11306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":22609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1932},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5717","reported_seq":5888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.775625+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:48.775625+0000","last_peered":"2024-04-22T08:28:48.775625+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:48.775625+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23384064,"num_objects":5709,"num_object_clones":0,"num_object_copies":11418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":7792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1872},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5704","reported_seq":5777,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.775616+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:48.775616+0000","last_peered":"2024-04-22T08:28:48.775616+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:48.775616+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23348434,"num_objects":5702,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":22802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1997},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5773","reported_seq":5914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:48.818410+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:48.818410+0000","last_peered":"2024-04-22T08:28:48.818410+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:48.818410+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23599524,"num_objects":5765,"num_object_clones":0,"num_object_copies":11530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":23051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1884},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":375824213,"num_objects":91765,"num_object_clones":0,"num_object_copies":183530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":91765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65367,"num_write_kb":261381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":4911721,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":762130432,"data_stored":762043457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91892,"ondisk_log_size":91892,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117018,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463100,"kb_used_data":305140,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93908740,"statfs":{"total":96636764160,"available":96162549760,"internally_reserved":0,"allocated":312463360,"data_stored":312262345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920781,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255524,"kb_used_data":103324,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":94116316,"statfs":{"total":96636764160,"available":96375107584,"internally_reserved":0,"allocated":105803776,"data_stored":105526223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":155840204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313245,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515388,"kb_used_data":335348,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93856452,"statfs":{"total":96636764160,"available":96109006848,"internally_reserved":0,"allocated":343396352,"data_stored":343201799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":184348371},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":4.202},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":343564288,"data_stored":343560809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105730048,"data_stored":105650031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312836096,"data_stored":312832617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:52.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:52.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'5774", 'reported_seq': 5834, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.845144+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:48.845144+0000', 'last_peered': '2024-04-22T08:28:48.845144+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:48.845144+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5774, 'log_dups_size': 0, 'ondisk_log_size': 5774, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23650304, 'num_objects': 5774, 'num_object_clones': 0, 'num_object_copies': 11548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5774, 'num_write_kb': 23096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1980}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'5708", 'reported_seq': 5828, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:49.577668+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:49.577668+0000', 'last_peered': '2024-04-22T08:28:49.577668+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:49.577668+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 5708, 'log_dups_size': 0, 'ondisk_log_size': 5708, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23363584, 'num_objects': 5704, 'num_object_clones': 0, 'num_object_copies': 11408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2535, 'num_write_kb': 10140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 1949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'5818", 'reported_seq': 5977, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.720781+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:48.720781+0000', 'last_peered': '2024-04-22T08:28:48.720781+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:48.720781+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 5818, 'log_dups_size': 0, 'ondisk_log_size': 5818, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23810048, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 11626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2687, 'num_write_kb': 10748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1934}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'5657", 'reported_seq': 8204, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:48.794473+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:48.794473+0000', 'last_peered': '2024-04-22T08:28:48.794473+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:48.794473+0000', 'last_undegraded': '2024-04-22T08:28:48.794473+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 5657, 'log_dups_size': 0, 'ondisk_log_size': 5657, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23146496, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 11302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2586, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2025, 'num_write_kb': 8100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1195, 'num_bytes_recovered': 4894720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3065}, {'shards': '0,2', 'objects': 2586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'5867", 'reported_seq': 5985, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:49.594991+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:49.594991+0000', 'last_peered': '2024-04-22T08:28:49.594991+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:49.594991+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 5867, 'log_dups_size': 0, 'ondisk_log_size': 5867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23766226, 'num_objects': 5804, 'num_object_clones': 0, 'num_object_copies': 11608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5807, 'num_write_kb': 23212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 1916}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:52.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'5746", 'reported_seq': 5820, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.834302+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:48.834302+0000', 'last_peered': '2024-04-22T08:28:48.834302+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:48.834302+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5746, 'log_dups_size': 0, 'ondisk_log_size': 5746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23504082, 'num_objects': 5740, 'num_object_clones': 0, 'num_object_copies': 11480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5744, 'num_write_kb': 22956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1969}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'5748", 'reported_seq': 5889, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:49.503867+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:49.503867+0000', 'last_peered': '2024-04-22T08:28:49.503867+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:49.503867+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 5748, 'log_dups_size': 0, 'ondisk_log_size': 5748, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23523328, 'num_objects': 5743, 'num_object_clones': 0, 'num_object_copies': 11486, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2202, 'num_write_kb': 8808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'5788", 'reported_seq': 5840, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:49.498242+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:49.498242+0000', 'last_peered': '2024-04-22T08:28:49.498242+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:49.498242+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5788, 'log_dups_size': 0, 'ondisk_log_size': 5788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23684306, 'num_objects': 5784, 'num_object_clones': 0, 'num_object_copies': 11568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5786, 'num_write_kb': 23131, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 1981}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:52.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'5654", 'reported_seq': 5730, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.795361+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:48.795361+0000', 'last_peered': '2024-04-22T08:28:48.795361+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:48.795361+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5654, 'log_dups_size': 0, 'ondisk_log_size': 5654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23151209, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 11306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5653, 'num_write_kb': 22609, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1932}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'5717", 'reported_seq': 5888, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.775625+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:48.775625+0000', 'last_peered': '2024-04-22T08:28:48.775625+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:48.775625+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 5717, 'log_dups_size': 0, 'ondisk_log_size': 5717, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23384064, 'num_objects': 5709, 'num_object_clones': 0, 'num_object_copies': 11418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1948, 'num_write_kb': 7792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1872}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'5704", 'reported_seq': 5777, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.775616+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:48.775616+0000', 'last_peered': '2024-04-22T08:28:48.775616+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:48.775616+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5704, 'log_dups_size': 0, 'ondisk_log_size': 5704, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23348434, 'num_objects': 5702, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5702, 'num_write_kb': 22802, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1997}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:52.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'5773", 'reported_seq': 5914, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:48.818410+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:48.818410+0000', 'last_peered': '2024-04-22T08:28:48.818410+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:48.818410+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 5773, 'log_dups_size': 0, 'ondisk_log_size': 5773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23599524, 'num_objects': 5765, 'num_object_clones': 0, 'num_object_copies': 11530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 23051, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1884}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:52.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:52.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:28:52.845026+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0213324 2024-04-22T08:28:52.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:28:52.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 100 20 93768 93748 3.6609 1.92578 0.0117697 0.0213324 2024-04-22T08:28:53.224 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:53.225 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/183530 objects degraded (22.778%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":168,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":91765,"data_bytes":375824213,"bytes_used":1263628288,"bytes_avail":288646664192,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":183530,"degraded_ratio":0.22777747507219531,"misplaced_objects":2586,"misplaced_total":183530,"misplaced_ratio":0.014090339454040212,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":82921,"recovering_keys_per_sec":0,"num_objects_recovered":243,"num_bytes_recovered":995328,"num_keys_recovered":0,"write_bytes_sec":1854291,"read_op_per_sec":0,"write_op_per_sec":452},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:28:53.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:28:53.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:53.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 101 20 94303 94283 3.64535 2.08984 0.0199444 0.0214225 2024-04-22T08:28:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:54.681 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:54.759 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-04-22T08:28:54.169832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":383012693,"num_objects":93520,"num_object_clones":0,"num_object_copies":187040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":93520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67122,"num_write_kb":268401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93647,"ondisk_log_size":93647,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1234080,"kb_used_data":752072,"kb_used_omap":12,"kb_used_meta":481971,"kb_avail":281881440,"statfs":{"total":289910292480,"available":288646594560,"internally_reserved":0,"allocated":770121728,"data_stored":769448607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":493538414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,6,13,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29446144,"num_objects":7189,"num_object_clones":0,"num_object_copies":14378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-374,"num_objects_unfound":0,"num_objects_dirty":7189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":28756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"2.f","version":"35'5971","reported_seq":6031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.770332+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:53.770332+0000","last_peered":"2024-04-22T08:28:53.770332+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:53.770332+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24457216,"num_objects":5971,"num_object_clones":0,"num_object_copies":11942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2177},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5779","reported_seq":5891,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.521811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:49.521811+0000","last_peered":"2024-04-22T08:28:49.521811+0000","last_clean":"2024-04-22T08:28:49.521811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:49.521811+0000","last_undegraded":"2024-04-22T08:28:49.521811+0000","last_fullsized":"2024-04-22T08:28:49.521811+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23646208,"num_objects":5773,"num_object_clones":0,"num_object_copies":11546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2835,"num_write_kb":11340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5708","reported_seq":5828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.577668+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:49.577668+0000","last_peered":"2024-04-22T08:28:49.577668+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:49.577668+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":5704,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":10140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":1949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5799","reported_seq":5826,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:49.590595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:49.590595+0000","last_peered":"2024-04-22T08:28:49.590595+0000","last_clean":"2024-04-22T08:28:49.590595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:49.590595+0000","last_undegraded":"2024-04-22T08:28:49.590595+0000","last_fullsized":"2024-04-22T08:28:49.590595+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752704,"num_objects":5799,"num_object_clones":0,"num_object_copies":11598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5885","reported_seq":6020,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.740541+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:53.740541+0000","last_peered":"2024-04-22T08:28:53.740541+0000","last_clean":"2024-04-22T08:28:53.740541+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:53.740541+0000","last_undegraded":"2024-04-22T08:28:53.740541+0000","last_fullsized":"2024-04-22T08:28:53.740541+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24076288,"num_objects":5878,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":11908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5998","reported_seq":6157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.789223+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:53.789223+0000","last_peered":"2024-04-22T08:28:53.789223+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:53.789223+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24547328,"num_objects":5993,"num_object_clones":0,"num_object_copies":11986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":11468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2114},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5835","reported_seq":8645,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:53.782731+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:53.782731+0000","last_peered":"2024-04-22T08:28:53.782731+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:53.782731+0000","last_undegraded":"2024-04-22T08:28:53.782731+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2203,"num_write_kb":8812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":5431296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3374},{"shards":"0,2","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5811","reported_seq":5952,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.816775+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:53.816775+0000","last_peered":"2024-04-22T08:28:53.816775+0000","last_clean":"2024-04-22T08:28:53.816775+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:53.816775+0000","last_undegraded":"2024-04-22T08:28:53.816775+0000","last_fullsized":"2024-04-22T08:28:53.816775+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23793664,"num_objects":5809,"num_object_clones":0,"num_object_copies":11618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":9836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'5867","reported_seq":5985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.594991+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:49.594991+0000","last_peered":"2024-04-22T08:28:49.594991+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:49.594991+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23766226,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":23212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5904","reported_seq":5979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.851456+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:53.851456+0000","last_peered":"2024-04-22T08:28:53.851456+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:53.851456+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5904,"log_dups_size":0,"ondisk_log_size":5904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24151250,"num_objects":5898,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5748","reported_seq":5889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.503867+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:49.503867+0000","last_peered":"2024-04-22T08:28:49.503867+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:49.503867+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23523328,"num_objects":5743,"num_object_clones":0,"num_object_copies":11486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5788","reported_seq":5840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:49.498242+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:49.498242+0000","last_peered":"2024-04-22T08:28:49.498242+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:49.498242+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23684306,"num_objects":5784,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":1981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5857","reported_seq":5933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.832328+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:53.832328+0000","last_peered":"2024-04-22T08:28:53.832328+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:53.832328+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23982697,"num_objects":5856,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5856,"num_write_kb":23421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5869","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.723691+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:53.723691+0000","last_peered":"2024-04-22T08:28:53.723691+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:53.723691+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24006656,"num_objects":5861,"num_object_clones":0,"num_object_copies":11722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2024},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5878","reported_seq":5951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.815773+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:53.815773+0000","last_peered":"2024-04-22T08:28:53.815773+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:53.815773+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5878,"log_dups_size":0,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24061138,"num_objects":5876,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2171},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5950","reported_seq":6090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.814483+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:53.814483+0000","last_peered":"2024-04-22T08:28:53.814483+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:53.814483+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24324516,"num_objects":5942,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":23759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2061},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":383012693,"num_objects":93520,"num_object_clones":0,"num_object_copies":187040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":93520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67122,"num_write_kb":268401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770666496,"data_stored":770579521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93647,"ondisk_log_size":93647,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117018,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463100,"kb_used_data":305140,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93908740,"statfs":{"total":96636764160,"available":96162549760,"internally_reserved":0,"allocated":312463360,"data_stored":312262345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920781,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255524,"kb_used_data":103324,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":94116316,"statfs":{"total":96636764160,"available":96375107584,"internally_reserved":0,"allocated":105803776,"data_stored":105526223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":155840204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313246,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515456,"kb_used_data":343608,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93856384,"statfs":{"total":96636764160,"available":96108937216,"internally_reserved":0,"allocated":351854592,"data_stored":351660039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":175959763},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.251},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":3.4830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352100352,"data_stored":352096873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105730048,"data_stored":105650031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312836096,"data_stored":312832617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:54.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 102 20 94869 94849 3.63129 2.21094 0.0164212 0.0215057 2024-04-22T08:28:55.843 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 103 20 95440 95420 3.61769 2.23047 0.063211 0.0215876 2024-04-22T08:28:56.845 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 104 20 96001 95981 3.60391 2.19141 0.0129376 0.0216706 2024-04-22T08:28:57.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:57.845 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 105 20 96569 96549 3.59072 2.21875 0.0375532 0.0217467 2024-04-22T08:28:58.449 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:58.449 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:58.530 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-22T08:28:58.170941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386871125,"num_objects":94462,"num_object_clones":0,"num_object_copies":188924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":94462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68064,"num_write_kb":272169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1244068,"kb_used_data":766156,"kb_used_omap":12,"kb_used_meta":477875,"kb_avail":281871452,"statfs":{"total":289910292480,"available":288636366848,"internally_reserved":0,"allocated":784543744,"data_stored":783870623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":489344110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,4,16,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22306816,"num_objects":5446,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-255,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":21784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003508"},"pg_stats":[{"pgid":"2.f","version":"35'5971","reported_seq":6031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.770332+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:53.770332+0000","last_peered":"2024-04-22T08:28:53.770332+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:53.770332+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24457216,"num_objects":5971,"num_object_clones":0,"num_object_copies":11942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2177},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5933","reported_seq":6045,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:54.591312+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:54.591312+0000","last_peered":"2024-04-22T08:28:54.591312+0000","last_clean":"2024-04-22T08:28:54.591312+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:54.591312+0000","last_undegraded":"2024-04-22T08:28:54.591312+0000","last_fullsized":"2024-04-22T08:28:54.591312+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24276992,"num_objects":5927,"num_object_clones":0,"num_object_copies":11854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5858","reported_seq":5977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.563699+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:54.563699+0000","last_peered":"2024-04-22T08:28:54.563699+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:54.563699+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23977984,"num_objects":5854,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2685,"num_write_kb":10740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5979","reported_seq":6006,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:54.612388+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:54.612388+0000","last_peered":"2024-04-22T08:28:54.612388+0000","last_clean":"2024-04-22T08:28:54.612388+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:54.612388+0000","last_undegraded":"2024-04-22T08:28:54.612388+0000","last_fullsized":"2024-04-22T08:28:54.612388+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24489984,"num_objects":5979,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":23916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5885","reported_seq":6020,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.740541+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:53.740541+0000","last_peered":"2024-04-22T08:28:53.740541+0000","last_clean":"2024-04-22T08:28:53.740541+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:53.740541+0000","last_undegraded":"2024-04-22T08:28:53.740541+0000","last_fullsized":"2024-04-22T08:28:53.740541+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24076288,"num_objects":5878,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":11908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5998","reported_seq":6157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.789223+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:53.789223+0000","last_peered":"2024-04-22T08:28:53.789223+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:53.789223+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24547328,"num_objects":5993,"num_object_clones":0,"num_object_copies":11986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":11468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2114},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5835","reported_seq":8645,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:53.782731+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:53.782731+0000","last_peered":"2024-04-22T08:28:53.782731+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:53.782731+0000","last_undegraded":"2024-04-22T08:28:53.782731+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2203,"num_write_kb":8812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":5431296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3374},{"shards":"0,2","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5811","reported_seq":5952,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.816775+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:53.816775+0000","last_peered":"2024-04-22T08:28:53.816775+0000","last_clean":"2024-04-22T08:28:53.816775+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:53.816775+0000","last_undegraded":"2024-04-22T08:28:53.816775+0000","last_fullsized":"2024-04-22T08:28:53.816775+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23793664,"num_objects":5809,"num_object_clones":0,"num_object_copies":11618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":9836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6024","reported_seq":6143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.627114+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:54.627114+0000","last_peered":"2024-04-22T08:28:54.627114+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:54.627114+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409298,"num_objects":5961,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5964,"num_write_kb":23840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5904","reported_seq":5979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.851456+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:53.851456+0000","last_peered":"2024-04-22T08:28:53.851456+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:53.851456+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5904,"log_dups_size":0,"ondisk_log_size":5904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24151250,"num_objects":5898,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5897","reported_seq":6040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.489733+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:54.489733+0000","last_peered":"2024-04-22T08:28:54.489733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:54.489733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24133632,"num_objects":5892,"num_object_clones":0,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":9404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2142}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5940","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.607801+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:54.607801+0000","last_peered":"2024-04-22T08:28:54.607801+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:54.607801+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24306898,"num_objects":5936,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":23739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5857","reported_seq":5933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.832328+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:53.832328+0000","last_peered":"2024-04-22T08:28:53.832328+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:53.832328+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23982697,"num_objects":5856,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5856,"num_write_kb":23421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5869","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.723691+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:53.723691+0000","last_peered":"2024-04-22T08:28:53.723691+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:53.723691+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24006656,"num_objects":5861,"num_object_clones":0,"num_object_copies":11722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2024},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5878","reported_seq":5951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.815773+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:53.815773+0000","last_peered":"2024-04-22T08:28:53.815773+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:53.815773+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5878,"log_dups_size":0,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24061138,"num_objects":5876,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2171},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5950","reported_seq":6090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.814483+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:53.814483+0000","last_peered":"2024-04-22T08:28:53.814483+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:53.814483+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24324516,"num_objects":5942,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":23759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2061},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":386871125,"num_objects":94462,"num_object_clones":0,"num_object_copies":188924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":94462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68064,"num_write_kb":272169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785055744,"data_stored":784968769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117019,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472524,"kb_used_data":310468,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93899316,"statfs":{"total":96636764160,"available":96152899584,"internally_reserved":0,"allocated":317919232,"data_stored":317718217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165932751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.33600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920782,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256088,"kb_used_data":112080,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94115752,"statfs":{"total":96636764160,"available":96374530048,"internally_reserved":0,"allocated":114769920,"data_stored":114492367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":147451596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":5.4539999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":3.0870000000000002},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":6.2969999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313246,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515456,"kb_used_data":343608,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93856384,"statfs":{"total":96636764160,"available":96108937216,"internally_reserved":0,"allocated":351854592,"data_stored":351660039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":175959763},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.251},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":3.4830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352100352,"data_stored":352096873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114831360,"data_stored":114751343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318124032,"data_stored":318120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:58.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:28:58.845 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 106 20 97094 97074 3.57619 2.05078 0.0136136 0.0218387 2024-04-22T08:28:59.418 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:28:59.419 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:28:59.491 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-22T08:28:58.170941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386871125,"num_objects":94462,"num_object_clones":0,"num_object_copies":188924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":94462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68064,"num_write_kb":272169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1244068,"kb_used_data":766156,"kb_used_omap":12,"kb_used_meta":477875,"kb_avail":281871452,"statfs":{"total":289910292480,"available":288636366848,"internally_reserved":0,"allocated":784543744,"data_stored":783870623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":489344110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,4,16,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22306816,"num_objects":5446,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-255,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":21784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003508"},"pg_stats":[{"pgid":"2.f","version":"35'5971","reported_seq":6031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.770332+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:53.770332+0000","last_peered":"2024-04-22T08:28:53.770332+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:53.770332+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24457216,"num_objects":5971,"num_object_clones":0,"num_object_copies":11942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2177},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'5933","reported_seq":6045,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:54.591312+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:54.591312+0000","last_peered":"2024-04-22T08:28:54.591312+0000","last_clean":"2024-04-22T08:28:54.591312+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:54.591312+0000","last_undegraded":"2024-04-22T08:28:54.591312+0000","last_fullsized":"2024-04-22T08:28:54.591312+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24276992,"num_objects":5927,"num_object_clones":0,"num_object_copies":11854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'5858","reported_seq":5977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.563699+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:54.563699+0000","last_peered":"2024-04-22T08:28:54.563699+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:54.563699+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23977984,"num_objects":5854,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2685,"num_write_kb":10740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'5979","reported_seq":6006,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:54.612388+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:54.612388+0000","last_peered":"2024-04-22T08:28:54.612388+0000","last_clean":"2024-04-22T08:28:54.612388+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:54.612388+0000","last_undegraded":"2024-04-22T08:28:54.612388+0000","last_fullsized":"2024-04-22T08:28:54.612388+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24489984,"num_objects":5979,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":23916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'5885","reported_seq":6020,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.740541+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:53.740541+0000","last_peered":"2024-04-22T08:28:53.740541+0000","last_clean":"2024-04-22T08:28:53.740541+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:53.740541+0000","last_undegraded":"2024-04-22T08:28:53.740541+0000","last_fullsized":"2024-04-22T08:28:53.740541+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24076288,"num_objects":5878,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":11908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'5998","reported_seq":6157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.789223+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:53.789223+0000","last_peered":"2024-04-22T08:28:53.789223+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:53.789223+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24547328,"num_objects":5993,"num_object_clones":0,"num_object_copies":11986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":11468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2114},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'5835","reported_seq":8645,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:53.782731+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:53.782731+0000","last_peered":"2024-04-22T08:28:53.782731+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:53.782731+0000","last_undegraded":"2024-04-22T08:28:53.782731+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2203,"num_write_kb":8812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":5431296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3374},{"shards":"0,2","objects":2455}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5811","reported_seq":5952,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.816775+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:53.816775+0000","last_peered":"2024-04-22T08:28:53.816775+0000","last_clean":"2024-04-22T08:28:53.816775+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:53.816775+0000","last_undegraded":"2024-04-22T08:28:53.816775+0000","last_fullsized":"2024-04-22T08:28:53.816775+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23793664,"num_objects":5809,"num_object_clones":0,"num_object_copies":11618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":9836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6024","reported_seq":6143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.627114+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:54.627114+0000","last_peered":"2024-04-22T08:28:54.627114+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:54.627114+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409298,"num_objects":5961,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5964,"num_write_kb":23840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'5904","reported_seq":5979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.851456+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:53.851456+0000","last_peered":"2024-04-22T08:28:53.851456+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:53.851456+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5904,"log_dups_size":0,"ondisk_log_size":5904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24151250,"num_objects":5898,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'5897","reported_seq":6040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.489733+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:54.489733+0000","last_peered":"2024-04-22T08:28:54.489733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:54.489733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24133632,"num_objects":5892,"num_object_clones":0,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":9404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2142}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'5940","reported_seq":5992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:54.607801+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:54.607801+0000","last_peered":"2024-04-22T08:28:54.607801+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:54.607801+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24306898,"num_objects":5936,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":23739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'5857","reported_seq":5933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.832328+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:53.832328+0000","last_peered":"2024-04-22T08:28:53.832328+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:53.832328+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23982697,"num_objects":5856,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5856,"num_write_kb":23421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'5869","reported_seq":6041,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.723691+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:53.723691+0000","last_peered":"2024-04-22T08:28:53.723691+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:53.723691+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24006656,"num_objects":5861,"num_object_clones":0,"num_object_copies":11722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2024},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'5878","reported_seq":5951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.815773+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:53.815773+0000","last_peered":"2024-04-22T08:28:53.815773+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:53.815773+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":5878,"log_dups_size":0,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24061138,"num_objects":5876,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2171},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'5950","reported_seq":6090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:53.814483+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:53.814483+0000","last_peered":"2024-04-22T08:28:53.814483+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:53.814483+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24324516,"num_objects":5942,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":23759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2061},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":386871125,"num_objects":94462,"num_object_clones":0,"num_object_copies":188924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2455,"num_objects_unfound":0,"num_objects_dirty":94462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68064,"num_write_kb":272169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":5448297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785055744,"data_stored":784968769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117019,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472524,"kb_used_data":310468,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93899316,"statfs":{"total":96636764160,"available":96152899584,"internally_reserved":0,"allocated":317919232,"data_stored":317718217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165932751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.33600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920782,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256088,"kb_used_data":112080,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94115752,"statfs":{"total":96636764160,"available":96374530048,"internally_reserved":0,"allocated":114769920,"data_stored":114492367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":147451596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":5.4539999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":3.0870000000000002},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":6.2969999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313246,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515456,"kb_used_data":343608,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93856384,"statfs":{"total":96636764160,"available":96108937216,"internally_reserved":0,"allocated":351854592,"data_stored":351660039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":175959763},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.251},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":3.4830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352100352,"data_stored":352096873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114831360,"data_stored":114751343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318124032,"data_stored":318120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:28:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:28:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'5971", 'reported_seq': 6031, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.770332+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:28:53.770332+0000', 'last_peered': '2024-04-22T08:28:53.770332+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:28:53.770332+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5971, 'log_dups_size': 0, 'ondisk_log_size': 5971, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24457216, 'num_objects': 5971, 'num_object_clones': 0, 'num_object_copies': 11942, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5971, 'num_write_kb': 23884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2177}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:28:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'5858", 'reported_seq': 5977, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:54.563699+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:54.563699+0000', 'last_peered': '2024-04-22T08:28:54.563699+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:54.563699+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 5858, 'log_dups_size': 0, 'ondisk_log_size': 5858, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23977984, 'num_objects': 5854, 'num_object_clones': 0, 'num_object_copies': 11708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2685, 'num_write_kb': 10740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'5998", 'reported_seq': 6157, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.789223+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:28:53.789223+0000', 'last_peered': '2024-04-22T08:28:53.789223+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:28:53.789223+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 5998, 'log_dups_size': 0, 'ondisk_log_size': 5998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24547328, 'num_objects': 5993, 'num_object_clones': 0, 'num_object_copies': 11986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2867, 'num_write_kb': 11468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2114}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'5835", 'reported_seq': 8645, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:28:53.782731+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:28:53.782731+0000', 'last_peered': '2024-04-22T08:28:53.782731+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:28:53.782731+0000', 'last_undegraded': '2024-04-22T08:28:53.782731+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 5835, 'log_dups_size': 0, 'ondisk_log_size': 5835, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23875584, 'num_objects': 5829, 'num_object_clones': 0, 'num_object_copies': 11658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2455, 'num_objects_unfound': 0, 'num_objects_dirty': 5829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2203, 'num_write_kb': 8812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1326, 'num_bytes_recovered': 5431296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3374}, {'shards': '0,2', 'objects': 2455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'6024", 'reported_seq': 6143, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:54.627114+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:54.627114+0000', 'last_peered': '2024-04-22T08:28:54.627114+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:54.627114+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 6024, 'log_dups_size': 0, 'ondisk_log_size': 6024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24409298, 'num_objects': 5961, 'num_object_clones': 0, 'num_object_copies': 11922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5964, 'num_write_kb': 23840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 2073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'5904", 'reported_seq': 5979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.851456+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:28:53.851456+0000', 'last_peered': '2024-04-22T08:28:53.851456+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:28:53.851456+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5904, 'log_dups_size': 0, 'ondisk_log_size': 5904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24151250, 'num_objects': 5898, 'num_object_clones': 0, 'num_object_copies': 11796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5902, 'num_write_kb': 23588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2127}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'5897", 'reported_seq': 6040, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:54.489733+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:54.489733+0000', 'last_peered': '2024-04-22T08:28:54.489733+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:54.489733+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 5897, 'log_dups_size': 0, 'ondisk_log_size': 5897, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24133632, 'num_objects': 5892, 'num_object_clones': 0, 'num_object_copies': 11784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2351, 'num_write_kb': 9404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 2142}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:59.493 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'5940", 'reported_seq': 5992, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:54.607801+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:54.607801+0000', 'last_peered': '2024-04-22T08:28:54.607801+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:54.607801+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5940, 'log_dups_size': 0, 'ondisk_log_size': 5940, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24306898, 'num_objects': 5936, 'num_object_clones': 0, 'num_object_copies': 11872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5938, 'num_write_kb': 23739, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 2133}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'5857", 'reported_seq': 5933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.832328+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:28:53.832328+0000', 'last_peered': '2024-04-22T08:28:53.832328+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:28:53.832328+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5857, 'log_dups_size': 0, 'ondisk_log_size': 5857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23982697, 'num_objects': 5856, 'num_object_clones': 0, 'num_object_copies': 11712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5856, 'num_write_kb': 23421, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2135}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'5869", 'reported_seq': 6041, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.723691+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:28:53.723691+0000', 'last_peered': '2024-04-22T08:28:53.723691+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:28:53.723691+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 5869, 'log_dups_size': 0, 'ondisk_log_size': 5869, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24006656, 'num_objects': 5861, 'num_object_clones': 0, 'num_object_copies': 11722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2100, 'num_write_kb': 8400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2024}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'5878", 'reported_seq': 5951, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.815773+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:28:53.815773+0000', 'last_peered': '2024-04-22T08:28:53.815773+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:28:53.815773+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 5878, 'log_dups_size': 0, 'ondisk_log_size': 5878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24061138, 'num_objects': 5876, 'num_object_clones': 0, 'num_object_copies': 11752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5876, 'num_write_kb': 23498, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2171}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:28:59.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'5950", 'reported_seq': 6090, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:53.814483+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:28:53.814483+0000', 'last_peered': '2024-04-22T08:28:53.814483+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:28:53.814483+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 5950, 'log_dups_size': 0, 'ondisk_log_size': 5950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24324516, 'num_objects': 5942, 'num_object_clones': 0, 'num_object_copies': 11884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5947, 'num_write_kb': 23759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2061}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:28:59.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:28:59.845 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 107 20 97619 97599 3.56194 2.05078 0.0150701 0.0219252 2024-04-22T08:29:00.304 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:00.304 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/188924 objects degraded (22.127%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":175,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":94462,"data_bytes":386871125,"bytes_used":1273925632,"bytes_avail":288636366848,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":188924,"degraded_ratio":0.22127416315555462,"misplaced_objects":2455,"misplaced_total":188924,"misplaced_ratio":0.012994643348648133,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":87014,"recovering_keys_per_sec":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"write_bytes_sec":1858358,"read_op_per_sec":0,"write_op_per_sec":453},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:00.335 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:00.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:00.845 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 108 20 98083 98063 3.54574 1.8125 0.0401317 0.0220249 2024-04-22T08:29:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:01.182 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:01.231 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-22T08:29:00.171415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398315349,"num_objects":97256,"num_object_clones":0,"num_object_copies":194512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2320,"num_objects_unfound":0,"num_objects_dirty":97256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70858,"num_write_kb":283345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1481,"num_bytes_recovered":6001257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97383,"ondisk_log_size":97383,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1292432,"kb_used_data":780088,"kb_used_omap":12,"kb_used_meta":512307,"kb_avail":281823088,"statfs":{"total":289910292480,"available":288586842112,"internally_reserved":0,"allocated":798810112,"data_stored":798136991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":524602478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,6,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-390,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.f","version":"35'6155","reported_seq":6215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.814787+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:28:58.814787+0000","last_peered":"2024-04-22T08:28:58.814787+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:28:58.814787+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25210880,"num_objects":6155,"num_object_clones":0,"num_object_copies":12310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6155,"num_write_kb":24620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2361},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6119","reported_seq":6231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.622871+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:59.622871+0000","last_peered":"2024-04-22T08:28:59.622871+0000","last_clean":"2024-04-22T08:28:59.622871+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:59.622871+0000","last_undegraded":"2024-04-22T08:28:59.622871+0000","last_fullsized":"2024-04-22T08:28:59.622871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25038848,"num_objects":6113,"num_object_clones":0,"num_object_copies":12226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":12700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6033","reported_seq":6153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.600728+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:59.600728+0000","last_peered":"2024-04-22T08:28:59.600728+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:59.600728+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24694784,"num_objects":6029,"num_object_clones":0,"num_object_copies":12058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":11440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6157","reported_seq":6183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.624043+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:59.624043+0000","last_peered":"2024-04-22T08:28:59.624043+0000","last_clean":"2024-04-22T08:28:59.624043+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:59.624043+0000","last_undegraded":"2024-04-22T08:28:59.624043+0000","last_fullsized":"2024-04-22T08:28:59.624043+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6076","reported_seq":6211,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:58.809387+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:28:58.809387+0000","last_peered":"2024-04-22T08:28:58.809387+0000","last_clean":"2024-04-22T08:28:58.809387+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:28:58.809387+0000","last_undegraded":"2024-04-22T08:28:58.809387+0000","last_fullsized":"2024-04-22T08:28:58.809387+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24858624,"num_objects":6069,"num_object_clones":0,"num_object_copies":12138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3168,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6174","reported_seq":6333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.747039+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:28:58.747039+0000","last_peered":"2024-04-22T08:28:58.747039+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:28:58.747039+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25268224,"num_objects":6169,"num_object_clones":0,"num_object_copies":12338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":12172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2290},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6001","reported_seq":9082,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:28:58.844395+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:28:58.844395+0000","last_peered":"2024-04-22T08:28:58.844395+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:28:58.844395+0000","last_undegraded":"2024-04-22T08:28:58.844395+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2320,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1461,"num_bytes_recovered":5984256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3675},{"shards":"0,2","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'5992","reported_seq":6134,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:58.859930+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:28:58.859930+0000","last_peered":"2024-04-22T08:28:58.859930+0000","last_clean":"2024-04-22T08:28:58.859930+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:28:58.859930+0000","last_undegraded":"2024-04-22T08:28:58.859930+0000","last_fullsized":"2024-04-22T08:28:58.859930+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24535040,"num_objects":5990,"num_object_clones":0,"num_object_copies":11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2640,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6200","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.646357+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:59.646357+0000","last_peered":"2024-04-22T08:28:59.646357+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:59.646357+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25130194,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6075","reported_seq":6148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.839078+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:28:58.839078+0000","last_peered":"2024-04-22T08:28:58.839078+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:28:58.839078+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24851666,"num_objects":6069,"num_object_clones":0,"num_object_copies":12138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":24272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2298},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6076","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.556289+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:59.556289+0000","last_peered":"2024-04-22T08:28:59.556289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:59.556289+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24866816,"num_objects":6071,"num_object_clones":0,"num_object_copies":12142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6094","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.522901+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:59.522901+0000","last_peered":"2024-04-22T08:28:59.522901+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:59.522901+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24937682,"num_objects":6090,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":24355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6025","reported_seq":6101,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.837213+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:28:58.837213+0000","last_peered":"2024-04-22T08:28:58.837213+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:28:58.837213+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24670825,"num_objects":6024,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":24093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2303},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6022","reported_seq":6194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.790614+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:28:58.790614+0000","last_peered":"2024-04-22T08:28:58.790614+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:28:58.790614+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2253,"num_write_kb":9012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2177},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6061","reported_seq":6134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.796234+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:28:58.796234+0000","last_peered":"2024-04-22T08:28:58.796234+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:28:58.796234+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6061,"log_dups_size":0,"ondisk_log_size":6061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810706,"num_objects":6059,"num_object_clones":0,"num_object_copies":12118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6059,"num_write_kb":24230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2354},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6123","reported_seq":6264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:58.827483+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:28:58.827483+0000","last_peered":"2024-04-22T08:28:58.827483+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:28:58.827483+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25033124,"num_objects":6115,"num_object_clones":0,"num_object_copies":12230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":24451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2234},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":398315349,"num_objects":97256,"num_object_clones":0,"num_object_copies":194512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2320,"num_objects_unfound":0,"num_objects_dirty":97256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70858,"num_write_kb":283345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1481,"num_bytes_recovered":6001257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799490048,"data_stored":799403073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97383,"ondisk_log_size":97383,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117020,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504296,"kb_used_data":316000,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93867544,"statfs":{"total":96636764160,"available":96120365056,"internally_reserved":0,"allocated":323584000,"data_stored":323382985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192802511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.254},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920782,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256088,"kb_used_data":112080,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94115752,"statfs":{"total":96636764160,"available":96374530048,"internally_reserved":0,"allocated":114769920,"data_stored":114492367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":147451596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":5.4539999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":3.0870000000000002},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":6.2969999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313247,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532048,"kb_used_data":352008,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93839792,"statfs":{"total":96636764160,"available":96091947008,"internally_reserved":0,"allocated":360456192,"data_stored":360261639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":184348371},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360751104,"data_stored":360747625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114831360,"data_stored":114751343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323907584,"data_stored":323904105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:01.848 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 109 20 98639 98619 3.53305 2.17188 0.0464108 0.022103 2024-04-22T08:29:02.848 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 110 20 99200 99180 3.52085 2.19141 0.0238492 0.0221817 2024-04-22T08:29:03.849 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 111 20 99756 99736 3.5087 2.17188 0.100625 0.0222576 2024-04-22T08:29:04.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:04.849 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 112 20 100271 100251 3.49531 2.01172 0.0298752 0.0223435 2024-04-22T08:29:04.993 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:05.083 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-22T08:29:04.172482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405139285,"num_objects":98922,"num_object_clones":0,"num_object_copies":197844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":98922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72524,"num_write_kb":290009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99049,"ondisk_log_size":99049,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1321264,"kb_used_data":796632,"kb_used_omap":12,"kb_used_meta":524595,"kb_avail":281794256,"statfs":{"total":289910292480,"available":288557318144,"internally_reserved":0,"allocated":815751168,"data_stored":815078047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":537185390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,3,11,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29315072,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-421,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7157,"num_write_kb":28628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":1724416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003097"},"pg_stats":[{"pgid":"2.f","version":"35'6331","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.889561+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:03.889561+0000","last_peered":"2024-04-22T08:29:03.889561+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:03.889561+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25931776,"num_objects":6331,"num_object_clones":0,"num_object_copies":12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":25324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2537},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6119","reported_seq":6231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.622871+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:59.622871+0000","last_peered":"2024-04-22T08:28:59.622871+0000","last_clean":"2024-04-22T08:28:59.622871+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:59.622871+0000","last_undegraded":"2024-04-22T08:28:59.622871+0000","last_fullsized":"2024-04-22T08:28:59.622871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25038848,"num_objects":6113,"num_object_clones":0,"num_object_copies":12226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":12700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6033","reported_seq":6153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.600728+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:59.600728+0000","last_peered":"2024-04-22T08:28:59.600728+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:59.600728+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24694784,"num_objects":6029,"num_object_clones":0,"num_object_copies":12058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":11440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6157","reported_seq":6183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.624043+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:59.624043+0000","last_peered":"2024-04-22T08:28:59.624043+0000","last_clean":"2024-04-22T08:28:59.624043+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:59.624043+0000","last_undegraded":"2024-04-22T08:28:59.624043+0000","last_fullsized":"2024-04-22T08:28:59.624043+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6259","reported_seq":6394,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.878971+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:03.878971+0000","last_peered":"2024-04-22T08:29:03.878971+0000","last_clean":"2024-04-22T08:29:03.878971+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:03.878971+0000","last_undegraded":"2024-04-22T08:29:03.878971+0000","last_fullsized":"2024-04-22T08:29:03.878971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":13404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6331","reported_seq":6489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.783624+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:03.783624+0000","last_peered":"2024-04-22T08:29:03.783624+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:03.783624+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25911296,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2447},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6158","reported_seq":9549,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:03.844777+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:03.844777+0000","last_peered":"2024-04-22T08:29:03.844777+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:03.844777+0000","last_undegraded":"2024-04-22T08:29:03.844777+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1616,"num_bytes_recovered":6619136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3987},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6159","reported_seq":6301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.887169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:03.887169+0000","last_peered":"2024-04-22T08:29:03.887169+0000","last_clean":"2024-04-22T08:29:03.887169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:03.887169+0000","last_undegraded":"2024-04-22T08:29:03.887169+0000","last_fullsized":"2024-04-22T08:29:03.887169+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6200","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.646357+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:59.646357+0000","last_peered":"2024-04-22T08:28:59.646357+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:59.646357+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25130194,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6247","reported_seq":6322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.809893+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:03.809893+0000","last_peered":"2024-04-22T08:29:03.809893+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:03.809893+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556178,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6076","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.556289+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:59.556289+0000","last_peered":"2024-04-22T08:28:59.556289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:59.556289+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24866816,"num_objects":6071,"num_object_clones":0,"num_object_copies":12142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6094","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.522901+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:59.522901+0000","last_peered":"2024-04-22T08:28:59.522901+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:59.522901+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24937682,"num_objects":6090,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":24355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6192","reported_seq":6268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.858383+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:03.858383+0000","last_peered":"2024-04-22T08:29:03.858383+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:03.858383+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25354857,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6171","reported_seq":6343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.868094+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:03.868094+0000","last_peered":"2024-04-22T08:29:03.868094+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:03.868094+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":9608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6236","reported_seq":6310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.891211+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:03.891211+0000","last_peered":"2024-04-22T08:29:03.891211+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:03.891211+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6236,"log_dups_size":0,"ondisk_log_size":6236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25527506,"num_objects":6234,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2529},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6286","reported_seq":6427,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.841107+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:03.841107+0000","last_peered":"2024-04-22T08:29:03.841107+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:03.841107+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25700772,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2397},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":405139285,"num_objects":98922,"num_object_clones":0,"num_object_copies":197844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":98922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72524,"num_write_kb":290009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":816455680,"data_stored":816368705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99049,"ondisk_log_size":99049,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117020,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504296,"kb_used_data":316000,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93867544,"statfs":{"total":96636764160,"available":96120365056,"internally_reserved":0,"allocated":323584000,"data_stored":323382985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192802511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.254},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920783,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272872,"kb_used_data":120672,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":94098968,"statfs":{"total":96636764160,"available":96357343232,"internally_reserved":0,"allocated":123568128,"data_stored":123290575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":155840204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.505},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.54900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313248,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544096,"kb_used_data":359960,"kb_used_omap":4,"kb_used_meta":184123,"kb_avail":93827744,"statfs":{"total":96636764160,"available":96079609856,"internally_reserved":0,"allocated":368599040,"data_stored":368404487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188542675},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":3.7890000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.22}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":3.8330000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":4.2060000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368951296,"data_stored":368947817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123596800,"data_stored":123516783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323907584,"data_stored":323904105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:05.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:05.849 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 113 20 100794 100774 3.48246 2.04297 0.0231321 0.0224243 2024-04-22T08:29:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:05.856 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:05.930 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-22T08:29:04.172482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405139285,"num_objects":98922,"num_object_clones":0,"num_object_copies":197844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":98922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72524,"num_write_kb":290009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99049,"ondisk_log_size":99049,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1321264,"kb_used_data":796632,"kb_used_omap":12,"kb_used_meta":524595,"kb_avail":281794256,"statfs":{"total":289910292480,"available":288557318144,"internally_reserved":0,"allocated":815751168,"data_stored":815078047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":537185390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,3,11,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29315072,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-421,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7157,"num_write_kb":28628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":1724416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003097"},"pg_stats":[{"pgid":"2.f","version":"35'6331","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.889561+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:03.889561+0000","last_peered":"2024-04-22T08:29:03.889561+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:03.889561+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25931776,"num_objects":6331,"num_object_clones":0,"num_object_copies":12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":25324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2537},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6119","reported_seq":6231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.622871+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:28:59.622871+0000","last_peered":"2024-04-22T08:28:59.622871+0000","last_clean":"2024-04-22T08:28:59.622871+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:28:59.622871+0000","last_undegraded":"2024-04-22T08:28:59.622871+0000","last_fullsized":"2024-04-22T08:28:59.622871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25038848,"num_objects":6113,"num_object_clones":0,"num_object_copies":12226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":12700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6033","reported_seq":6153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.600728+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:28:59.600728+0000","last_peered":"2024-04-22T08:28:59.600728+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:28:59.600728+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24694784,"num_objects":6029,"num_object_clones":0,"num_object_copies":12058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":11440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6157","reported_seq":6183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:59.624043+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:28:59.624043+0000","last_peered":"2024-04-22T08:28:59.624043+0000","last_clean":"2024-04-22T08:28:59.624043+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:28:59.624043+0000","last_undegraded":"2024-04-22T08:28:59.624043+0000","last_fullsized":"2024-04-22T08:28:59.624043+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6259","reported_seq":6394,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.878971+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:03.878971+0000","last_peered":"2024-04-22T08:29:03.878971+0000","last_clean":"2024-04-22T08:29:03.878971+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:03.878971+0000","last_undegraded":"2024-04-22T08:29:03.878971+0000","last_fullsized":"2024-04-22T08:29:03.878971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":13404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6331","reported_seq":6489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.783624+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:03.783624+0000","last_peered":"2024-04-22T08:29:03.783624+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:03.783624+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25911296,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2447},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6158","reported_seq":9549,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:03.844777+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:03.844777+0000","last_peered":"2024-04-22T08:29:03.844777+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:03.844777+0000","last_undegraded":"2024-04-22T08:29:03.844777+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1616,"num_bytes_recovered":6619136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3987},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6159","reported_seq":6301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.887169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:03.887169+0000","last_peered":"2024-04-22T08:29:03.887169+0000","last_clean":"2024-04-22T08:29:03.887169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:03.887169+0000","last_undegraded":"2024-04-22T08:29:03.887169+0000","last_fullsized":"2024-04-22T08:29:03.887169+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6200","reported_seq":6319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.646357+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:28:59.646357+0000","last_peered":"2024-04-22T08:28:59.646357+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:28:59.646357+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25130194,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6247","reported_seq":6322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.809893+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:03.809893+0000","last_peered":"2024-04-22T08:29:03.809893+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:03.809893+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556178,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6076","reported_seq":6219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.556289+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:28:59.556289+0000","last_peered":"2024-04-22T08:28:59.556289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:28:59.556289+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24866816,"num_objects":6071,"num_object_clones":0,"num_object_copies":12142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":10120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6094","reported_seq":6146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:28:59.522901+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:28:59.522901+0000","last_peered":"2024-04-22T08:28:59.522901+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:28:59.522901+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24937682,"num_objects":6090,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":24355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6192","reported_seq":6268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.858383+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:03.858383+0000","last_peered":"2024-04-22T08:29:03.858383+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:03.858383+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25354857,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6171","reported_seq":6343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.868094+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:03.868094+0000","last_peered":"2024-04-22T08:29:03.868094+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:03.868094+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":9608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6236","reported_seq":6310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.891211+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:03.891211+0000","last_peered":"2024-04-22T08:29:03.891211+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:03.891211+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6236,"log_dups_size":0,"ondisk_log_size":6236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25527506,"num_objects":6234,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2529},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6286","reported_seq":6427,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.841107+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:03.841107+0000","last_peered":"2024-04-22T08:29:03.841107+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:03.841107+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25700772,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2397},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":405139285,"num_objects":98922,"num_object_clones":0,"num_object_copies":197844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":98922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72524,"num_write_kb":290009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":816455680,"data_stored":816368705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99049,"ondisk_log_size":99049,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117020,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504296,"kb_used_data":316000,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93867544,"statfs":{"total":96636764160,"available":96120365056,"internally_reserved":0,"allocated":323584000,"data_stored":323382985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192802511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.254},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920783,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272872,"kb_used_data":120672,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":94098968,"statfs":{"total":96636764160,"available":96357343232,"internally_reserved":0,"allocated":123568128,"data_stored":123290575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":155840204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.505},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.54900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313248,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544096,"kb_used_data":359960,"kb_used_omap":4,"kb_used_meta":184123,"kb_avail":93827744,"statfs":{"total":96636764160,"available":96079609856,"internally_reserved":0,"allocated":368599040,"data_stored":368404487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188542675},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":3.7890000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.22}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":3.8330000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":4.2060000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368951296,"data_stored":368947817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123596800,"data_stored":123516783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323907584,"data_stored":323904105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'6331", 'reported_seq': 6392, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.889561+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:03.889561+0000', 'last_peered': '2024-04-22T08:29:03.889561+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:03.889561+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6331, 'log_dups_size': 0, 'ondisk_log_size': 6331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25931776, 'num_objects': 6331, 'num_object_clones': 0, 'num_object_copies': 12662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6331, 'num_write_kb': 25324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2537}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'6033", 'reported_seq': 6153, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:59.600728+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:28:59.600728+0000', 'last_peered': '2024-04-22T08:28:59.600728+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:28:59.600728+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 6033, 'log_dups_size': 0, 'ondisk_log_size': 6033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24694784, 'num_objects': 6029, 'num_object_clones': 0, 'num_object_copies': 12058, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2860, 'num_write_kb': 11440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 2274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'6331", 'reported_seq': 6489, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.783624+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:03.783624+0000', 'last_peered': '2024-04-22T08:29:03.783624+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:03.783624+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 6331, 'log_dups_size': 0, 'ondisk_log_size': 6331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25911296, 'num_objects': 6326, 'num_object_clones': 0, 'num_object_copies': 12652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3200, 'num_write_kb': 12800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2447}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'6158", 'reported_seq': 9549, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:03.844777+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:03.844777+0000', 'last_peered': '2024-04-22T08:29:03.844777+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:03.844777+0000', 'last_undegraded': '2024-04-22T08:29:03.844777+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 6158, 'log_dups_size': 0, 'ondisk_log_size': 6158, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25198592, 'num_objects': 6152, 'num_object_clones': 0, 'num_object_copies': 12304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2165, 'num_objects_unfound': 0, 'num_objects_dirty': 6152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2526, 'num_write_kb': 10104, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1616, 'num_bytes_recovered': 6619136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3987}, {'shards': '0,2', 'objects': 2165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'6200", 'reported_seq': 6319, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:59.646357+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:28:59.646357+0000', 'last_peered': '2024-04-22T08:28:59.646357+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:28:59.646357+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 6200, 'log_dups_size': 0, 'ondisk_log_size': 6200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25130194, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 12274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6140, 'num_write_kb': 24544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 2249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'6247", 'reported_seq': 6322, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.809893+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:03.809893+0000', 'last_peered': '2024-04-22T08:29:03.809893+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:03.809893+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6247, 'log_dups_size': 0, 'ondisk_log_size': 6247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25556178, 'num_objects': 6241, 'num_object_clones': 0, 'num_object_copies': 12482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6245, 'num_write_kb': 24960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2470}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'6076", 'reported_seq': 6219, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:59.556289+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:28:59.556289+0000', 'last_peered': '2024-04-22T08:28:59.556289+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:28:59.556289+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 6076, 'log_dups_size': 0, 'ondisk_log_size': 6076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24866816, 'num_objects': 6071, 'num_object_clones': 0, 'num_object_copies': 12142, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2530, 'num_write_kb': 10120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 2321}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'6094", 'reported_seq': 6146, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:28:59.522901+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:28:59.522901+0000', 'last_peered': '2024-04-22T08:28:59.522901+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:28:59.522901+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6094, 'log_dups_size': 0, 'ondisk_log_size': 6094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24937682, 'num_objects': 6090, 'num_object_clones': 0, 'num_object_copies': 12180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6092, 'num_write_kb': 24355, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 2287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:05.933 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'6192", 'reported_seq': 6268, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.858383+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:03.858383+0000', 'last_peered': '2024-04-22T08:29:03.858383+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:03.858383+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6192, 'log_dups_size': 0, 'ondisk_log_size': 6192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25354857, 'num_objects': 6191, 'num_object_clones': 0, 'num_object_copies': 12382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6191, 'num_write_kb': 24761, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2470}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'6171", 'reported_seq': 6343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.868094+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:03.868094+0000', 'last_peered': '2024-04-22T08:29:03.868094+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:03.868094+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 6171, 'log_dups_size': 0, 'ondisk_log_size': 6171, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25243648, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 12326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2402, 'num_write_kb': 9608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2326}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'6236", 'reported_seq': 6310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.891211+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:03.891211+0000', 'last_peered': '2024-04-22T08:29:03.891211+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:03.891211+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6236, 'log_dups_size': 0, 'ondisk_log_size': 6236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25527506, 'num_objects': 6234, 'num_object_clones': 0, 'num_object_copies': 12468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6234, 'num_write_kb': 24930, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2529}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:05.934 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'6286", 'reported_seq': 6427, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:03.841107+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:03.841107+0000', 'last_peered': '2024-04-22T08:29:03.841107+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:03.841107+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 6286, 'log_dups_size': 0, 'ondisk_log_size': 6286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25700772, 'num_objects': 6278, 'num_object_clones': 0, 'num_object_copies': 12556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6283, 'num_write_kb': 25103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2397}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:05.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:06.849 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 114 19 101304 101285 3.46943 1.99609 0.0391489 0.0225095 2024-04-22T08:29:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:06.875 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/199858 objects degraded (20.917%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":181,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":99929,"data_bytes":409263957,"bytes_used":1346002944,"bytes_avail":288564289536,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":199858,"degraded_ratio":0.20916850964184572,"misplaced_objects":2165,"misplaced_total":199858,"misplaced_ratio":0.010832691210759639,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":98960,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":2187029,"read_op_per_sec":0,"write_op_per_sec":533},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:06.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:06.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:07.651 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:07.653 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:07.706 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-04-22T08:29:06.172991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409263957,"num_objects":99929,"num_object_clones":0,"num_object_copies":199858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":99929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73531,"num_write_kb":294037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100056,"ondisk_log_size":100056,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1314456,"kb_used_data":802112,"kb_used_omap":12,"kb_used_meta":512307,"kb_avail":281801064,"statfs":{"total":289910292480,"available":288564289536,"internally_reserved":0,"allocated":821362688,"data_stored":820689567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":524602478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,3,13,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26251264,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-290,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6409,"num_write_kb":25636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003159"},"pg_stats":[{"pgid":"2.f","version":"35'6331","reported_seq":6392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.889561+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:03.889561+0000","last_peered":"2024-04-22T08:29:03.889561+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:03.889561+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25931776,"num_objects":6331,"num_object_clones":0,"num_object_copies":12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":25324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2537},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6302","reported_seq":6414,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:04.619291+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:04.619291+0000","last_peered":"2024-04-22T08:29:04.619291+0000","last_clean":"2024-04-22T08:29:04.619291+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:04.619291+0000","last_undegraded":"2024-04-22T08:29:04.619291+0000","last_fullsized":"2024-04-22T08:29:04.619291+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6302,"log_dups_size":0,"ondisk_log_size":6302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25788416,"num_objects":6296,"num_object_clones":0,"num_object_copies":12592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":13432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6202","reported_seq":6322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:04.581659+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:04.581659+0000","last_peered":"2024-04-22T08:29:04.581659+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:04.581659+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25387008,"num_objects":6198,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":12116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2443}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6308","reported_seq":6335,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:04.635184+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:04.635184+0000","last_peered":"2024-04-22T08:29:04.635184+0000","last_clean":"2024-04-22T08:29:04.635184+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:04.635184+0000","last_undegraded":"2024-04-22T08:29:04.635184+0000","last_fullsized":"2024-04-22T08:29:04.635184+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25837568,"num_objects":6308,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":25232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6259","reported_seq":6394,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.878971+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:03.878971+0000","last_peered":"2024-04-22T08:29:03.878971+0000","last_clean":"2024-04-22T08:29:03.878971+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:03.878971+0000","last_undegraded":"2024-04-22T08:29:03.878971+0000","last_fullsized":"2024-04-22T08:29:03.878971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":13404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6331","reported_seq":6489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.783624+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:03.783624+0000","last_peered":"2024-04-22T08:29:03.783624+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:03.783624+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25911296,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2447},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6158","reported_seq":9549,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:03.844777+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:03.844777+0000","last_peered":"2024-04-22T08:29:03.844777+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:03.844777+0000","last_undegraded":"2024-04-22T08:29:03.844777+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":10104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1616,"num_bytes_recovered":6619136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3987},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6159","reported_seq":6301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:03.887169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:03.887169+0000","last_peered":"2024-04-22T08:29:03.887169+0000","last_clean":"2024-04-22T08:29:03.887169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:03.887169+0000","last_undegraded":"2024-04-22T08:29:03.887169+0000","last_fullsized":"2024-04-22T08:29:03.887169+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6369","reported_seq":6489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:04.605197+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:04.605197+0000","last_peered":"2024-04-22T08:29:04.605197+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:04.605197+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6369,"log_dups_size":0,"ondisk_log_size":6369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25822418,"num_objects":6306,"num_object_clones":0,"num_object_copies":12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":25220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.091683+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:55.091683+0000","last_peered":"2024-04-22T08:27:55.091683+0000","last_clean":"2024-04-22T08:27:55.091683+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:55.091683+0000","last_undegraded":"2024-04-22T08:27:55.091683+0000","last_fullsized":"2024-04-22T08:27:55.091683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6247","reported_seq":6322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.809893+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:03.809893+0000","last_peered":"2024-04-22T08:29:03.809893+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:03.809893+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556178,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6247","reported_seq":6390,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:04.664518+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:04.664518+0000","last_peered":"2024-04-22T08:29:04.664518+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:04.664518+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25567232,"num_objects":6242,"num_object_clones":0,"num_object_copies":12484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2701,"num_write_kb":10804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6258","reported_seq":6310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:04.610388+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:04.610388+0000","last_peered":"2024-04-22T08:29:04.610388+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:04.610388+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25609426,"num_objects":6254,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":25011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6192","reported_seq":6268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.858383+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:03.858383+0000","last_peered":"2024-04-22T08:29:03.858383+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:03.858383+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25354857,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6171","reported_seq":6343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.868094+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:03.868094+0000","last_peered":"2024-04-22T08:29:03.868094+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:03.868094+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":9608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T08:27:55.092790+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:55.092790+0000","last_peered":"2024-04-22T08:27:55.092790+0000","last_clean":"2024-04-22T08:27:55.092790+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:55.092790+0000","last_undegraded":"2024-04-22T08:27:55.092790+0000","last_fullsized":"2024-04-22T08:27:55.092790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6236","reported_seq":6310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.891211+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:03.891211+0000","last_peered":"2024-04-22T08:29:03.891211+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:03.891211+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6236,"log_dups_size":0,"ondisk_log_size":6236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25527506,"num_objects":6234,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2529},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6286","reported_seq":6427,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:03.841107+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:03.841107+0000","last_peered":"2024-04-22T08:29:03.841107+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:03.841107+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25700772,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2397},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":409263957,"num_objects":99929,"num_object_clones":0,"num_object_copies":199858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":99929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73531,"num_write_kb":294037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":6636137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":821964800,"data_stored":821877825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100056,"ondisk_log_size":100056,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117021,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497488,"kb_used_data":321480,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93874352,"statfs":{"total":96636764160,"available":96127336448,"internally_reserved":0,"allocated":329195520,"data_stored":328994505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":4.0839999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":2.5369999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.219},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":2.4849999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920783,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272872,"kb_used_data":120672,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":94098968,"statfs":{"total":96636764160,"available":96357343232,"internally_reserved":0,"allocated":123568128,"data_stored":123290575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":155840204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.505},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.54900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313248,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544096,"kb_used_data":359960,"kb_used_omap":4,"kb_used_meta":184123,"kb_avail":93827744,"statfs":{"total":96636764160,"available":96079609856,"internally_reserved":0,"allocated":368599040,"data_stored":368404487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":188542675},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":3.7890000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.22}]},{"osd":2,"last update":"Mon Apr 22 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":3.8330000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5030000000000001,"5min":3.5030000000000001,"15min":3.5030000000000001},"last":4.2060000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368951296,"data_stored":368947817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123596800,"data_stored":123516783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329416704,"data_stored":329413225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:07.853 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 115 20 101828 101808 3.45702 2.04297 0.0306881 0.0225905 2024-04-22T08:29:08.849 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 116 20 102377 102357 3.44571 2.14453 0.00606198 0.0226612 2024-04-22T08:29:09.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 117 20 102925 102905 3.43456 2.14062 0.0108188 0.0227367 2024-04-22T08:29:10.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:10.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 118 20 103461 103441 3.4232 2.09375 0.0317076 0.0228138 2024-04-22T08:29:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:11.500 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:11.565 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-04-22T08:29:10.179373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420147029,"num_objects":102586,"num_object_clones":0,"num_object_copies":205172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":102586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76188,"num_write_kb":304665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":7193193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102713,"ondisk_log_size":102713,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1337820,"kb_used_data":823812,"kb_used_omap":12,"kb_used_meta":513971,"kb_avail":281777700,"statfs":{"total":289910292480,"available":288540364800,"internally_reserved":0,"allocated":843583488,"data_stored":842910367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":526306414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,3,3,7,9,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33275904,"num_objects":8124,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-426,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008432"},"pg_stats":[{"pgid":"2.f","version":"35'6461","reported_seq":6522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.814339+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:08.814339+0000","last_peered":"2024-04-22T08:29:08.814339+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:08.814339+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26464256,"num_objects":6461,"num_object_clones":0,"num_object_copies":12922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":25844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2667},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6456","reported_seq":6568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.595832+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:09.595832+0000","last_peered":"2024-04-22T08:29:09.595832+0000","last_clean":"2024-04-22T08:29:09.595832+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:09.595832+0000","last_undegraded":"2024-04-22T08:29:09.595832+0000","last_fullsized":"2024-04-22T08:29:09.595832+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26419200,"num_objects":6450,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":14048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6384","reported_seq":6504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.696229+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:09.696229+0000","last_peered":"2024-04-22T08:29:09.696229+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:09.696229+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26132480,"num_objects":6380,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":12844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6483","reported_seq":6510,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.665127+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:09.665127+0000","last_peered":"2024-04-22T08:29:09.665127+0000","last_clean":"2024-04-22T08:29:09.665127+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:09.665127+0000","last_undegraded":"2024-04-22T08:29:09.665127+0000","last_fullsized":"2024-04-22T08:29:09.665127+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6409","reported_seq":6545,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:08.867225+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:08.867225+0000","last_peered":"2024-04-22T08:29:08.867225+0000","last_clean":"2024-04-22T08:29:08.867225+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:08.867225+0000","last_undegraded":"2024-04-22T08:29:08.867225+0000","last_fullsized":"2024-04-22T08:29:08.867225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26222592,"num_objects":6402,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":14004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6525","reported_seq":6681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.895873+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:08.895873+0000","last_peered":"2024-04-22T08:29:08.895873+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:08.895873+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6525,"log_dups_size":0,"ondisk_log_size":6525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705920,"num_objects":6520,"num_object_clones":0,"num_object_copies":13040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":13576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2641},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6322","reported_seq":9990,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:08.907013+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:08.907013+0000","last_peered":"2024-04-22T08:29:08.907013+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:08.907013+0000","last_undegraded":"2024-04-22T08:29:08.907013+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25870336,"num_objects":6316,"num_object_clones":0,"num_object_copies":12632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":10760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4287},{"shards":"0,2","objects":2029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6323","reported_seq":6465,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:08.851071+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:08.851071+0000","last_peered":"2024-04-22T08:29:08.851071+0000","last_clean":"2024-04-22T08:29:08.851071+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:08.851071+0000","last_undegraded":"2024-04-22T08:29:08.851071+0000","last_fullsized":"2024-04-22T08:29:08.851071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25890816,"num_objects":6321,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6552","reported_seq":6672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.649821+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:09.649821+0000","last_peered":"2024-04-22T08:29:09.649821+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:09.649821+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26571986,"num_objects":6489,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6492,"num_write_kb":25952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6412","reported_seq":6484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.933565+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:08.933565+0000","last_peered":"2024-04-22T08:29:08.933565+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:08.933565+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6412,"log_dups_size":0,"ondisk_log_size":6412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26232018,"num_objects":6406,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6410,"num_write_kb":25620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2635},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6410","reported_seq":6553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.679222+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:09.679222+0000","last_peered":"2024-04-22T08:29:09.679222+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:09.679222+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26234880,"num_objects":6405,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6416","reported_seq":6469,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.590765+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:09.590765+0000","last_peered":"2024-04-22T08:29:09.590765+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:09.590765+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26256594,"num_objects":6412,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6366","reported_seq":6442,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.853459+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:08.853459+0000","last_peered":"2024-04-22T08:29:08.853459+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:08.853459+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26067561,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":25457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2644},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6319","reported_seq":6491,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.900965+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:08.900965+0000","last_peered":"2024-04-22T08:29:08.900965+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:08.900965+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25849856,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":10200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2474},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6402","reported_seq":6476,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.897020+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:08.897020+0000","last_peered":"2024-04-22T08:29:08.897020+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:08.897020+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26207442,"num_objects":6400,"num_object_clones":0,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":25594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2695},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6473","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.851585+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:08.851585+0000","last_peered":"2024-04-22T08:29:08.851585+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:08.851585+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26466724,"num_objects":6465,"num_object_clones":0,"num_object_copies":12930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":25851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2584},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":420147029,"num_objects":102586,"num_object_clones":0,"num_object_copies":205172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":102586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76188,"num_write_kb":304665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":7193193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":844390400,"data_stored":844310383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102713,"ondisk_log_size":102713,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117022,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506800,"kb_used_data":326696,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93865040,"statfs":{"total":96636764160,"available":96117800960,"internally_reserved":0,"allocated":334536704,"data_stored":334335689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184413903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":5.4409999999999998},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.77400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.657},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23100000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920784,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269200,"kb_used_data":129288,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94102640,"statfs":{"total":96636764160,"available":96361103360,"internally_reserved":0,"allocated":132390912,"data_stored":132113359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":143257292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":1.143}]}]},{"osd":0,"up_from":21,"seq":90194313249,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561820,"kb_used_data":367828,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93810020,"statfs":{"total":96636764160,"available":96061460480,"internally_reserved":0,"allocated":376655872,"data_stored":376461319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":198635219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,0,4,4,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":3.0710000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":2.1800000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":3.2349999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377024512,"data_stored":377021033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132460544,"data_stored":132387485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334905344,"data_stored":334901865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:11.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:11.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 119 20 103979 103959 3.41143 2.02344 0.0225883 0.0228917 2024-04-22T08:29:12.439 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:12.440 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:12.497 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-04-22T08:29:12.179915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420147029,"num_objects":102586,"num_object_clones":0,"num_object_copies":205172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":102586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76188,"num_write_kb":304665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":7193193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102713,"ondisk_log_size":102713,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1337820,"kb_used_data":823812,"kb_used_omap":12,"kb_used_meta":513971,"kb_avail":281777700,"statfs":{"total":289910292480,"available":288540364800,"internally_reserved":0,"allocated":843583488,"data_stored":842910367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":526306414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,3,3,7,9,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21831680,"num_objects":5330,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-291,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5330,"num_write_kb":21320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":1191936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008499"},"pg_stats":[{"pgid":"2.f","version":"35'6461","reported_seq":6522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.814339+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:08.814339+0000","last_peered":"2024-04-22T08:29:08.814339+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:08.814339+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26464256,"num_objects":6461,"num_object_clones":0,"num_object_copies":12922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":25844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2667},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6456","reported_seq":6568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.595832+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:09.595832+0000","last_peered":"2024-04-22T08:29:09.595832+0000","last_clean":"2024-04-22T08:29:09.595832+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:09.595832+0000","last_undegraded":"2024-04-22T08:29:09.595832+0000","last_fullsized":"2024-04-22T08:29:09.595832+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26419200,"num_objects":6450,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":14048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6384","reported_seq":6504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.696229+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:09.696229+0000","last_peered":"2024-04-22T08:29:09.696229+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:09.696229+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26132480,"num_objects":6380,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":12844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6483","reported_seq":6510,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.665127+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:09.665127+0000","last_peered":"2024-04-22T08:29:09.665127+0000","last_clean":"2024-04-22T08:29:09.665127+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:09.665127+0000","last_undegraded":"2024-04-22T08:29:09.665127+0000","last_fullsized":"2024-04-22T08:29:09.665127+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6409","reported_seq":6545,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:08.867225+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:08.867225+0000","last_peered":"2024-04-22T08:29:08.867225+0000","last_clean":"2024-04-22T08:29:08.867225+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:08.867225+0000","last_undegraded":"2024-04-22T08:29:08.867225+0000","last_fullsized":"2024-04-22T08:29:08.867225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26222592,"num_objects":6402,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":14004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6525","reported_seq":6681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.895873+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:08.895873+0000","last_peered":"2024-04-22T08:29:08.895873+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:08.895873+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6525,"log_dups_size":0,"ondisk_log_size":6525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705920,"num_objects":6520,"num_object_clones":0,"num_object_copies":13040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":13576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2641},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6322","reported_seq":9990,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:08.907013+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:08.907013+0000","last_peered":"2024-04-22T08:29:08.907013+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:08.907013+0000","last_undegraded":"2024-04-22T08:29:08.907013+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25870336,"num_objects":6316,"num_object_clones":0,"num_object_copies":12632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":10760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4287},{"shards":"0,2","objects":2029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6323","reported_seq":6465,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:08.851071+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:08.851071+0000","last_peered":"2024-04-22T08:29:08.851071+0000","last_clean":"2024-04-22T08:29:08.851071+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:08.851071+0000","last_undegraded":"2024-04-22T08:29:08.851071+0000","last_fullsized":"2024-04-22T08:29:08.851071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25890816,"num_objects":6321,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6552","reported_seq":6672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.649821+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:09.649821+0000","last_peered":"2024-04-22T08:29:09.649821+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:09.649821+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26571986,"num_objects":6489,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6492,"num_write_kb":25952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6412","reported_seq":6484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.933565+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:08.933565+0000","last_peered":"2024-04-22T08:29:08.933565+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:08.933565+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6412,"log_dups_size":0,"ondisk_log_size":6412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26232018,"num_objects":6406,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6410,"num_write_kb":25620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2635},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6410","reported_seq":6553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.679222+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:09.679222+0000","last_peered":"2024-04-22T08:29:09.679222+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:09.679222+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26234880,"num_objects":6405,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6416","reported_seq":6469,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.590765+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:09.590765+0000","last_peered":"2024-04-22T08:29:09.590765+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:09.590765+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26256594,"num_objects":6412,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6366","reported_seq":6442,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.853459+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:08.853459+0000","last_peered":"2024-04-22T08:29:08.853459+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:08.853459+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26067561,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":25457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2644},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6319","reported_seq":6491,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.900965+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:08.900965+0000","last_peered":"2024-04-22T08:29:08.900965+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:08.900965+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25849856,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":10200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2474},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6402","reported_seq":6476,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.897020+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:08.897020+0000","last_peered":"2024-04-22T08:29:08.897020+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:08.897020+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26207442,"num_objects":6400,"num_object_clones":0,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":25594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2695},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6473","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:08.851585+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:08.851585+0000","last_peered":"2024-04-22T08:29:08.851585+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:08.851585+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26466724,"num_objects":6465,"num_object_clones":0,"num_object_copies":12930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":25851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2584},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":420147029,"num_objects":102586,"num_object_clones":0,"num_object_copies":205172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":2029,"num_objects_unfound":0,"num_objects_dirty":102586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76188,"num_write_kb":304665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1774,"num_bytes_recovered":7193193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":844390400,"data_stored":844310383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102713,"ondisk_log_size":102713,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117022,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506800,"kb_used_data":326696,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93865040,"statfs":{"total":96636764160,"available":96117800960,"internally_reserved":0,"allocated":334536704,"data_stored":334335689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184413903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":5.4409999999999998},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.77400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.657},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23100000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920784,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269200,"kb_used_data":129288,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94102640,"statfs":{"total":96636764160,"available":96361103360,"internally_reserved":0,"allocated":132390912,"data_stored":132113359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":143257292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":1.143}]}]},{"osd":0,"up_from":21,"seq":90194313249,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561820,"kb_used_data":367828,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93810020,"statfs":{"total":96636764160,"available":96061460480,"internally_reserved":0,"allocated":376655872,"data_stored":376461319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":198635219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,0,4,4,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":3.0710000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":2.1800000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":3.2349999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377024512,"data_stored":377021033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132460544,"data_stored":132387485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334905344,"data_stored":334901865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'6461", 'reported_seq': 6522, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.814339+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:08.814339+0000', 'last_peered': '2024-04-22T08:29:08.814339+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:08.814339+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6461, 'log_dups_size': 0, 'ondisk_log_size': 6461, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26464256, 'num_objects': 6461, 'num_object_clones': 0, 'num_object_copies': 12922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6461, 'num_write_kb': 25844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2667}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'6384", 'reported_seq': 6504, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:09.696229+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:09.696229+0000', 'last_peered': '2024-04-22T08:29:09.696229+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:09.696229+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 6384, 'log_dups_size': 0, 'ondisk_log_size': 6384, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26132480, 'num_objects': 6380, 'num_object_clones': 0, 'num_object_copies': 12760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3211, 'num_write_kb': 12844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 2625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'6525", 'reported_seq': 6681, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.895873+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:08.895873+0000', 'last_peered': '2024-04-22T08:29:08.895873+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:08.895873+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 6525, 'log_dups_size': 0, 'ondisk_log_size': 6525, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26705920, 'num_objects': 6520, 'num_object_clones': 0, 'num_object_copies': 13040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3394, 'num_write_kb': 13576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2641}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'6322", 'reported_seq': 9990, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:08.907013+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:08.907013+0000', 'last_peered': '2024-04-22T08:29:08.907013+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:08.907013+0000', 'last_undegraded': '2024-04-22T08:29:08.907013+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 6322, 'log_dups_size': 0, 'ondisk_log_size': 6322, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25870336, 'num_objects': 6316, 'num_object_clones': 0, 'num_object_copies': 12632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2029, 'num_objects_unfound': 0, 'num_objects_dirty': 6316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2690, 'num_write_kb': 10760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1754, 'num_bytes_recovered': 7176192, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4287}, {'shards': '0,2', 'objects': 2029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'6552", 'reported_seq': 6672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:09.649821+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:09.649821+0000', 'last_peered': '2024-04-22T08:29:09.649821+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:09.649821+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 6552, 'log_dups_size': 0, 'ondisk_log_size': 6552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26571986, 'num_objects': 6489, 'num_object_clones': 0, 'num_object_copies': 12978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6492, 'num_write_kb': 25952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 2601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'6412", 'reported_seq': 6484, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.933565+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:08.933565+0000', 'last_peered': '2024-04-22T08:29:08.933565+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:08.933565+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6412, 'log_dups_size': 0, 'ondisk_log_size': 6412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26232018, 'num_objects': 6406, 'num_object_clones': 0, 'num_object_copies': 12812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6410, 'num_write_kb': 25620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2635}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'6410", 'reported_seq': 6553, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:09.679222+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:09.679222+0000', 'last_peered': '2024-04-22T08:29:09.679222+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:09.679222+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 6410, 'log_dups_size': 0, 'ondisk_log_size': 6410, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26234880, 'num_objects': 6405, 'num_object_clones': 0, 'num_object_copies': 12810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2864, 'num_write_kb': 11456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 2655}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'6416", 'reported_seq': 6469, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:09.590765+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:09.590765+0000', 'last_peered': '2024-04-22T08:29:09.590765+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:09.590765+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6416, 'log_dups_size': 0, 'ondisk_log_size': 6416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26256594, 'num_objects': 6412, 'num_object_clones': 0, 'num_object_copies': 12824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6414, 'num_write_kb': 25643, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 2609}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:12.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'6366", 'reported_seq': 6442, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.853459+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:08.853459+0000', 'last_peered': '2024-04-22T08:29:08.853459+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:08.853459+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26067561, 'num_objects': 6365, 'num_object_clones': 0, 'num_object_copies': 12730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 25457, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2644}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'6319", 'reported_seq': 6491, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.900965+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:08.900965+0000', 'last_peered': '2024-04-22T08:29:08.900965+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:08.900965+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 6319, 'log_dups_size': 0, 'ondisk_log_size': 6319, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25849856, 'num_objects': 6311, 'num_object_clones': 0, 'num_object_copies': 12622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2550, 'num_write_kb': 10200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2474}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'6402", 'reported_seq': 6476, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.897020+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:08.897020+0000', 'last_peered': '2024-04-22T08:29:08.897020+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:08.897020+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6402, 'log_dups_size': 0, 'ondisk_log_size': 6402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26207442, 'num_objects': 6400, 'num_object_clones': 0, 'num_object_copies': 12800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6400, 'num_write_kb': 25594, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2695}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:12.504 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'6473", 'reported_seq': 6614, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:08.851585+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:08.851585+0000', 'last_peered': '2024-04-22T08:29:08.851585+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:08.851585+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 6473, 'log_dups_size': 0, 'ondisk_log_size': 6473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26466724, 'num_objects': 6465, 'num_object_clones': 0, 'num_object_copies': 12930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6470, 'num_write_kb': 25851, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2584}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:12.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:12.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:29:12.852106+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0229698 2024-04-22T08:29:12.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:29:12.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 120 20 104482 104462 3.39938 1.96484 0.0175989 0.0229698 2024-04-22T08:29:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:13.436 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/205172 objects degraded (20.375%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":102586,"data_bytes":420147029,"bytes_used":1369927680,"bytes_avail":288540364800,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":205172,"degraded_ratio":0.20375099916167896,"misplaced_objects":2029,"misplaced_total":205172,"misplaced_ratio":0.009889263642212388,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":99257,"recovering_keys_per_sec":0,"num_objects_recovered":293,"num_bytes_recovered":1191936,"num_keys_recovered":0,"write_bytes_sec":1818018,"read_op_per_sec":0,"write_op_per_sec":443},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:13.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:13.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:13.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 121 20 104992 104972 3.38775 1.99219 0.054336 0.0230529 2024-04-22T08:29:14.310 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:14.313 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:14.360 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-04-22T08:29:14.180423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426954581,"num_objects":104248,"num_object_clones":0,"num_object_copies":208496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":104248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77850,"num_write_kb":311313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104375,"ondisk_log_size":104375,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1370720,"kb_used_data":840328,"kb_used_omap":12,"kb_used_meta":530355,"kb_avail":281744800,"statfs":{"total":289910292480,"available":288506675200,"internally_reserved":0,"allocated":860495872,"data_stored":859829709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":543083630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,1,8,9,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28639232,"num_objects":6992,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":6992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008451"},"pg_stats":[{"pgid":"2.f","version":"35'6642","reported_seq":6703,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.889292+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:13.889292+0000","last_peered":"2024-04-22T08:29:13.889292+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:13.889292+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27205632,"num_objects":6642,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":26568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2848},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6456","reported_seq":6568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.595832+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:09.595832+0000","last_peered":"2024-04-22T08:29:09.595832+0000","last_clean":"2024-04-22T08:29:09.595832+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:09.595832+0000","last_undegraded":"2024-04-22T08:29:09.595832+0000","last_fullsized":"2024-04-22T08:29:09.595832+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26419200,"num_objects":6450,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":14048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6384","reported_seq":6504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.696229+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:09.696229+0000","last_peered":"2024-04-22T08:29:09.696229+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:09.696229+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26132480,"num_objects":6380,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":12844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6483","reported_seq":6510,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.665127+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:09.665127+0000","last_peered":"2024-04-22T08:29:09.665127+0000","last_clean":"2024-04-22T08:29:09.665127+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:09.665127+0000","last_undegraded":"2024-04-22T08:29:09.665127+0000","last_fullsized":"2024-04-22T08:29:09.665127+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6558","reported_seq":6694,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.968772+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:13.968772+0000","last_peered":"2024-04-22T08:29:13.968772+0000","last_clean":"2024-04-22T08:29:13.968772+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:13.968772+0000","last_undegraded":"2024-04-22T08:29:13.968772+0000","last_fullsized":"2024-04-22T08:29:13.968772+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26832896,"num_objects":6551,"num_object_clones":0,"num_object_copies":13102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":14600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6685","reported_seq":6843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.915456+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:13.915456+0000","last_peered":"2024-04-22T08:29:13.915456+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:13.915456+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6685,"log_dups_size":0,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3554,"num_write_kb":14216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2801},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6498","reported_seq":10429,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:13.986181+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:13.986181+0000","last_peered":"2024-04-22T08:29:13.986181+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:13.986181+0000","last_undegraded":"2024-04-22T08:29:13.986181+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":6492,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":7720960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6484","reported_seq":6626,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.933626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:13.933626+0000","last_peered":"2024-04-22T08:29:13.933626+0000","last_clean":"2024-04-22T08:29:13.933626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:13.933626+0000","last_undegraded":"2024-04-22T08:29:13.933626+0000","last_fullsized":"2024-04-22T08:29:13.933626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26550272,"num_objects":6482,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6552","reported_seq":6672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.649821+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:09.649821+0000","last_peered":"2024-04-22T08:29:09.649821+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:09.649821+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26571986,"num_objects":6489,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6492,"num_write_kb":25952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6570","reported_seq":6644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.974490+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:13.974490+0000","last_peered":"2024-04-22T08:29:13.974490+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:13.974490+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26879186,"num_objects":6564,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":26252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6410","reported_seq":6553,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.679222+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:09.679222+0000","last_peered":"2024-04-22T08:29:09.679222+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:09.679222+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26234880,"num_objects":6405,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6416","reported_seq":6469,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:09.590765+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:09.590765+0000","last_peered":"2024-04-22T08:29:09.590765+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:09.590765+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26256594,"num_objects":6412,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6537","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.956838+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:13.956838+0000","last_peered":"2024-04-22T08:29:13.956838+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:13.956838+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6537,"log_dups_size":0,"ondisk_log_size":6537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767977,"num_objects":6536,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":26141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2815},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6505","reported_seq":6677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.995309+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:13.995309+0000","last_peered":"2024-04-22T08:29:13.995309+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:13.995309+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26611712,"num_objects":6497,"num_object_clones":0,"num_object_copies":12994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2660},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6562","reported_seq":6636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.941883+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:13.941883+0000","last_peered":"2024-04-22T08:29:13.941883+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:13.941883+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26862802,"num_objects":6560,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2855},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6633","reported_seq":6773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.957987+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:13.957987+0000","last_peered":"2024-04-22T08:29:13.957987+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:13.957987+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27122084,"num_objects":6625,"num_object_clones":0,"num_object_copies":13250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":26491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2744},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":426954581,"num_objects":104248,"num_object_clones":0,"num_object_copies":208496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":104248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77850,"num_write_kb":311313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":861413376,"data_stored":861333359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104375,"ondisk_log_size":104375,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117022,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506800,"kb_used_data":326696,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93865040,"statfs":{"total":96636764160,"available":96117800960,"internally_reserved":0,"allocated":334536704,"data_stored":334335689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184413903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":5.4409999999999998},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.77400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.657},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.23100000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920785,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286104,"kb_used_data":138000,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94085736,"statfs":{"total":96636764160,"available":96343793664,"internally_reserved":0,"allocated":141312000,"data_stored":141041405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":151645900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.7769999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.625},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":3.3769999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313250,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577816,"kb_used_data":375632,"kb_used_omap":4,"kb_used_meta":202171,"kb_avail":93794024,"statfs":{"total":96636764160,"available":96045080576,"internally_reserved":0,"allocated":384647168,"data_stored":384452615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":207023827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.8900000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":2.9300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":3.1030000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385114112,"data_stored":385110633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141393920,"data_stored":141320861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":334905344,"data_stored":334901865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:14.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 122 20 105518 105498 3.37683 2.05469 0.0485381 0.0231276 2024-04-22T08:29:15.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 123 20 106059 106039 3.36656 2.11328 0.0759134 0.0231959 2024-04-22T08:29:16.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 124 20 106597 106577 3.35635 2.10156 0.0103076 0.0232679 2024-04-22T08:29:17.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:17.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 125 20 107111 107091 3.34557 2.00781 0.0418947 0.0233432 2024-04-22T08:29:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:18.052 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:18.134 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-04-22T08:29:16.181188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":430907221,"num_objects":105213,"num_object_clones":0,"num_object_copies":210426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":105213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78815,"num_write_kb":315173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105340,"ondisk_log_size":105340,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1379972,"kb_used_data":845484,"kb_used_omap":12,"kb_used_meta":534451,"kb_avail":281735548,"statfs":{"total":289910292480,"available":288497201152,"internally_reserved":0,"allocated":865775616,"data_stored":865109453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":547277934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,3,2,7,8,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25767936,"num_objects":6291,"num_object_clones":0,"num_object_copies":12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":25164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":1101824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008705"},"pg_stats":[{"pgid":"2.f","version":"35'6642","reported_seq":6703,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.889292+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:13.889292+0000","last_peered":"2024-04-22T08:29:13.889292+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:13.889292+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27205632,"num_objects":6642,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":26568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2848},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6599","reported_seq":6711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:14.642900+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:14.642900+0000","last_peered":"2024-04-22T08:29:14.642900+0000","last_clean":"2024-04-22T08:29:14.642900+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:14.642900+0000","last_undegraded":"2024-04-22T08:29:14.642900+0000","last_fullsized":"2024-04-22T08:29:14.642900+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27004928,"num_objects":6593,"num_object_clones":0,"num_object_copies":13186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":14620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6542","reported_seq":6662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.672786+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:14.672786+0000","last_peered":"2024-04-22T08:29:14.672786+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:14.672786+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26779648,"num_objects":6538,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":13476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6659","reported_seq":6686,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:14.668513+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:14.668513+0000","last_peered":"2024-04-22T08:29:14.668513+0000","last_clean":"2024-04-22T08:29:14.668513+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:14.668513+0000","last_undegraded":"2024-04-22T08:29:14.668513+0000","last_fullsized":"2024-04-22T08:29:14.668513+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":26636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6558","reported_seq":6694,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.968772+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:13.968772+0000","last_peered":"2024-04-22T08:29:13.968772+0000","last_clean":"2024-04-22T08:29:13.968772+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:13.968772+0000","last_undegraded":"2024-04-22T08:29:13.968772+0000","last_fullsized":"2024-04-22T08:29:13.968772+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26832896,"num_objects":6551,"num_object_clones":0,"num_object_copies":13102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":14600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6685","reported_seq":6843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.915456+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:13.915456+0000","last_peered":"2024-04-22T08:29:13.915456+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:13.915456+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6685,"log_dups_size":0,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3554,"num_write_kb":14216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2801},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6498","reported_seq":10429,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:13.986181+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:13.986181+0000","last_peered":"2024-04-22T08:29:13.986181+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:13.986181+0000","last_undegraded":"2024-04-22T08:29:13.986181+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":6492,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":7720960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6484","reported_seq":6626,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.933626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:13.933626+0000","last_peered":"2024-04-22T08:29:13.933626+0000","last_clean":"2024-04-22T08:29:13.933626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:13.933626+0000","last_undegraded":"2024-04-22T08:29:13.933626+0000","last_fullsized":"2024-04-22T08:29:13.933626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26550272,"num_objects":6482,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6730","reported_seq":6850,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.680082+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:14.680082+0000","last_peered":"2024-04-22T08:29:14.680082+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:14.680082+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27301074,"num_objects":6667,"num_object_clones":0,"num_object_copies":13334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6570","reported_seq":6644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.974490+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:13.974490+0000","last_peered":"2024-04-22T08:29:13.974490+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:13.974490+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26879186,"num_objects":6564,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":26252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6563","reported_seq":6707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.652800+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:14.652800+0000","last_peered":"2024-04-22T08:29:14.652800+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:14.652800+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26861568,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":12068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2808}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6573","reported_seq":6626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.663680+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:14.663680+0000","last_peered":"2024-04-22T08:29:14.663680+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:14.663680+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6573,"log_dups_size":0,"ondisk_log_size":6573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26899666,"num_objects":6569,"num_object_clones":0,"num_object_copies":13138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":26271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6537","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.956838+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:13.956838+0000","last_peered":"2024-04-22T08:29:13.956838+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:13.956838+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6537,"log_dups_size":0,"ondisk_log_size":6537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767977,"num_objects":6536,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":26141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2815},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6505","reported_seq":6677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.995309+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:13.995309+0000","last_peered":"2024-04-22T08:29:13.995309+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:13.995309+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26611712,"num_objects":6497,"num_object_clones":0,"num_object_copies":12994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2660},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6562","reported_seq":6636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.941883+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:13.941883+0000","last_peered":"2024-04-22T08:29:13.941883+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:13.941883+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26862802,"num_objects":6560,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2855},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6633","reported_seq":6773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.957987+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:13.957987+0000","last_peered":"2024-04-22T08:29:13.957987+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:13.957987+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27122084,"num_objects":6625,"num_object_clones":0,"num_object_copies":13250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":26491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2744},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":430907221,"num_objects":105213,"num_object_clones":0,"num_object_copies":210426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":105213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78815,"num_write_kb":315173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":866623488,"data_stored":866543471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105340,"ondisk_log_size":105340,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516052,"kb_used_data":331852,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93855788,"statfs":{"total":96636764160,"available":96108326912,"internally_reserved":0,"allocated":339816448,"data_stored":339615433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.474},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":4.7569999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.26300000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920785,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286104,"kb_used_data":138000,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94085736,"statfs":{"total":96636764160,"available":96343793664,"internally_reserved":0,"allocated":141312000,"data_stored":141041405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":151645900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.7769999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.625},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":3.3769999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313250,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577816,"kb_used_data":375632,"kb_used_omap":4,"kb_used_meta":202171,"kb_avail":93794024,"statfs":{"total":96636764160,"available":96045080576,"internally_reserved":0,"allocated":384647168,"data_stored":384452615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":207023827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.8900000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":2.9300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":3.1030000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385114112,"data_stored":385110633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141393920,"data_stored":141320861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340115456,"data_stored":340111977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:18.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:18.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 126 20 107630 107610 3.33511 2.02734 0.0318273 0.0234164 2024-04-22T08:29:19.064 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:19.086 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:19.147 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-04-22T08:29:18.183691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":430907221,"num_objects":105213,"num_object_clones":0,"num_object_copies":210426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":105213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78815,"num_write_kb":315173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105340,"ondisk_log_size":105340,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398512,"kb_used_data":854168,"kb_used_omap":12,"kb_used_meta":544307,"kb_avail":281717008,"statfs":{"total":289910292480,"available":288478216192,"internally_reserved":0,"allocated":874668032,"data_stored":874001869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":557370478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,3,6,9,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21643264,"num_objects":5284,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5284,"num_write_kb":21136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":1101824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010700"},"pg_stats":[{"pgid":"2.f","version":"35'6642","reported_seq":6703,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.889292+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:13.889292+0000","last_peered":"2024-04-22T08:29:13.889292+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:13.889292+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27205632,"num_objects":6642,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":26568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2848},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6599","reported_seq":6711,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:14.642900+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:14.642900+0000","last_peered":"2024-04-22T08:29:14.642900+0000","last_clean":"2024-04-22T08:29:14.642900+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:14.642900+0000","last_undegraded":"2024-04-22T08:29:14.642900+0000","last_fullsized":"2024-04-22T08:29:14.642900+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27004928,"num_objects":6593,"num_object_clones":0,"num_object_copies":13186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":14620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6542","reported_seq":6662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.672786+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:14.672786+0000","last_peered":"2024-04-22T08:29:14.672786+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:14.672786+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26779648,"num_objects":6538,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":13476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2783}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6659","reported_seq":6686,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:14.668513+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:14.668513+0000","last_peered":"2024-04-22T08:29:14.668513+0000","last_clean":"2024-04-22T08:29:14.668513+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:14.668513+0000","last_undegraded":"2024-04-22T08:29:14.668513+0000","last_fullsized":"2024-04-22T08:29:14.668513+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":26636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6558","reported_seq":6694,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.968772+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:13.968772+0000","last_peered":"2024-04-22T08:29:13.968772+0000","last_clean":"2024-04-22T08:29:13.968772+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:13.968772+0000","last_undegraded":"2024-04-22T08:29:13.968772+0000","last_fullsized":"2024-04-22T08:29:13.968772+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26832896,"num_objects":6551,"num_object_clones":0,"num_object_copies":13102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":14600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6685","reported_seq":6843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.915456+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:13.915456+0000","last_peered":"2024-04-22T08:29:13.915456+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:13.915456+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6685,"log_dups_size":0,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3554,"num_write_kb":14216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2801},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6498","reported_seq":10429,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:13.986181+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:13.986181+0000","last_peered":"2024-04-22T08:29:13.986181+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:13.986181+0000","last_undegraded":"2024-04-22T08:29:13.986181+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":6492,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":7720960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6484","reported_seq":6626,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:13.933626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:13.933626+0000","last_peered":"2024-04-22T08:29:13.933626+0000","last_clean":"2024-04-22T08:29:13.933626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:13.933626+0000","last_undegraded":"2024-04-22T08:29:13.933626+0000","last_fullsized":"2024-04-22T08:29:13.933626+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26550272,"num_objects":6482,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6730","reported_seq":6850,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.680082+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:14.680082+0000","last_peered":"2024-04-22T08:29:14.680082+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:14.680082+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27301074,"num_objects":6667,"num_object_clones":0,"num_object_copies":13334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6570","reported_seq":6644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.974490+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:13.974490+0000","last_peered":"2024-04-22T08:29:13.974490+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:13.974490+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26879186,"num_objects":6564,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":26252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2793},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6563","reported_seq":6707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.652800+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:14.652800+0000","last_peered":"2024-04-22T08:29:14.652800+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:14.652800+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26861568,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":12068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2808}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6573","reported_seq":6626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:14.663680+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:14.663680+0000","last_peered":"2024-04-22T08:29:14.663680+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:14.663680+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6573,"log_dups_size":0,"ondisk_log_size":6573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26899666,"num_objects":6569,"num_object_clones":0,"num_object_copies":13138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":26271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6537","reported_seq":6614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.956838+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:13.956838+0000","last_peered":"2024-04-22T08:29:13.956838+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:13.956838+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6537,"log_dups_size":0,"ondisk_log_size":6537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767977,"num_objects":6536,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":26141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2815},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6505","reported_seq":6677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.995309+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:13.995309+0000","last_peered":"2024-04-22T08:29:13.995309+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:13.995309+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26611712,"num_objects":6497,"num_object_clones":0,"num_object_copies":12994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2660},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6562","reported_seq":6636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.941883+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:13.941883+0000","last_peered":"2024-04-22T08:29:13.941883+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:13.941883+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26862802,"num_objects":6560,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2855},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6633","reported_seq":6773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:13.957987+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:13.957987+0000","last_peered":"2024-04-22T08:29:13.957987+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:13.957987+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27122084,"num_objects":6625,"num_object_clones":0,"num_object_copies":13250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":26491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2744},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":430907221,"num_objects":105213,"num_object_clones":0,"num_object_copies":210426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1896,"num_objects_unfound":0,"num_objects_dirty":105213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78815,"num_write_kb":315173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":7737961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":875417600,"data_stored":875337583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105340,"ondisk_log_size":105340,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516052,"kb_used_data":331852,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93855788,"statfs":{"total":96636764160,"available":96108326912,"internally_reserved":0,"allocated":339816448,"data_stored":339615433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.474},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":4.7569999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.26300000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920786,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304644,"kb_used_data":146684,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94067196,"statfs":{"total":96636764160,"available":96324808704,"internally_reserved":0,"allocated":150204416,"data_stored":149933821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":161738444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":2.601}]}]},{"osd":0,"up_from":21,"seq":90194313250,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577816,"kb_used_data":375632,"kb_used_omap":4,"kb_used_meta":202171,"kb_avail":93794024,"statfs":{"total":96636764160,"available":96045080576,"internally_reserved":0,"allocated":384647168,"data_stored":384452615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":207023827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,3,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.8900000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":2.9300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":3.1030000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385114112,"data_stored":385110633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":150188032,"data_stored":150114973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340115456,"data_stored":340111977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:19.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:19.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'6642", 'reported_seq': 6703, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.889292+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:13.889292+0000', 'last_peered': '2024-04-22T08:29:13.889292+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:13.889292+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6642, 'log_dups_size': 0, 'ondisk_log_size': 6642, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27205632, 'num_objects': 6642, 'num_object_clones': 0, 'num_object_copies': 13284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6642, 'num_write_kb': 26568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2848}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:19.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'6542", 'reported_seq': 6662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:14.672786+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:14.672786+0000', 'last_peered': '2024-04-22T08:29:14.672786+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:14.672786+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 6542, 'log_dups_size': 0, 'ondisk_log_size': 6542, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26779648, 'num_objects': 6538, 'num_object_clones': 0, 'num_object_copies': 13076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3369, 'num_write_kb': 13476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 2783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:19.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'6685", 'reported_seq': 6843, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.915456+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:13.915456+0000', 'last_peered': '2024-04-22T08:29:13.915456+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:13.915456+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 6685, 'log_dups_size': 0, 'ondisk_log_size': 6685, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27361280, 'num_objects': 6680, 'num_object_clones': 0, 'num_object_copies': 13360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3554, 'num_write_kb': 14216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2801}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'6498", 'reported_seq': 10429, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:13.986181+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:13.986181+0000', 'last_peered': '2024-04-22T08:29:13.986181+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:13.986181+0000', 'last_undegraded': '2024-04-22T08:29:13.986181+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 6498, 'log_dups_size': 0, 'ondisk_log_size': 6498, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26591232, 'num_objects': 6492, 'num_object_clones': 0, 'num_object_copies': 12984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1896, 'num_objects_unfound': 0, 'num_objects_dirty': 6492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2866, 'num_write_kb': 11464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1887, 'num_bytes_recovered': 7720960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4596}, {'shards': '0,2', 'objects': 1896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'6730", 'reported_seq': 6850, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:14.680082+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:14.680082+0000', 'last_peered': '2024-04-22T08:29:14.680082+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:14.680082+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 6730, 'log_dups_size': 0, 'ondisk_log_size': 6730, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27301074, 'num_objects': 6667, 'num_object_clones': 0, 'num_object_copies': 13334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6670, 'num_write_kb': 26664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 2779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'6570", 'reported_seq': 6644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.974490+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:13.974490+0000', 'last_peered': '2024-04-22T08:29:13.974490+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:13.974490+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6570, 'log_dups_size': 0, 'ondisk_log_size': 6570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26879186, 'num_objects': 6564, 'num_object_clones': 0, 'num_object_copies': 13128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6568, 'num_write_kb': 26252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2793}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'6563", 'reported_seq': 6707, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:14.652800+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:14.652800+0000', 'last_peered': '2024-04-22T08:29:14.652800+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:14.652800+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 6563, 'log_dups_size': 0, 'ondisk_log_size': 6563, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26861568, 'num_objects': 6558, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3017, 'num_write_kb': 12068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 2808}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'6573", 'reported_seq': 6626, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:14.663680+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:14.663680+0000', 'last_peered': '2024-04-22T08:29:14.663680+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:14.663680+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6573, 'log_dups_size': 0, 'ondisk_log_size': 6573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26899666, 'num_objects': 6569, 'num_object_clones': 0, 'num_object_copies': 13138, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6571, 'num_write_kb': 26271, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 2766}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:19.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'6537", 'reported_seq': 6614, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.956838+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:13.956838+0000', 'last_peered': '2024-04-22T08:29:13.956838+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:13.956838+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6537, 'log_dups_size': 0, 'ondisk_log_size': 6537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26767977, 'num_objects': 6536, 'num_object_clones': 0, 'num_object_copies': 13072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6536, 'num_write_kb': 26141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2815}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'6505", 'reported_seq': 6677, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.995309+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:13.995309+0000', 'last_peered': '2024-04-22T08:29:13.995309+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:13.995309+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 6505, 'log_dups_size': 0, 'ondisk_log_size': 6505, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26611712, 'num_objects': 6497, 'num_object_clones': 0, 'num_object_copies': 12994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2736, 'num_write_kb': 10944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2660}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'6562", 'reported_seq': 6636, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.941883+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:13.941883+0000', 'last_peered': '2024-04-22T08:29:13.941883+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:13.941883+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6562, 'log_dups_size': 0, 'ondisk_log_size': 6562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26862802, 'num_objects': 6560, 'num_object_clones': 0, 'num_object_copies': 13120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6560, 'num_write_kb': 26234, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2855}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:19.150 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'6633", 'reported_seq': 6773, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:13.957987+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:13.957987+0000', 'last_peered': '2024-04-22T08:29:13.957987+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:13.957987+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 6633, 'log_dups_size': 0, 'ondisk_log_size': 6633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27122084, 'num_objects': 6625, 'num_object_clones': 0, 'num_object_copies': 13250, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6630, 'num_write_kb': 26491, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2744}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:19.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:19.850 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 127 20 108139 108119 3.3245 1.98828 0.0995358 0.0234918 2024-04-22T08:29:20.208 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:20.208 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/210426 objects degraded (19.866%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":195,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":105213,"data_bytes":430907221,"bytes_used":1432076288,"bytes_avail":288478216192,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":210426,"degraded_ratio":0.19866366323553172,"misplaced_objects":1896,"misplaced_total":210426,"misplaced_ratio":0.0090102934048073915,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":91736,"recovering_keys_per_sec":0,"num_objects_recovered":271,"num_bytes_recovered":1101824,"num_keys_recovered":0,"write_bytes_sec":1801998,"read_op_per_sec":0,"write_op_per_sec":439},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:20.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:20.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:20.853 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 128 20 108634 108614 3.31357 1.93359 0.117893 0.0235673 2024-04-22T08:29:21.060 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:21.060 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:21.123 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-04-22T08:29:20.184154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441802581,"num_objects":107873,"num_object_clones":0,"num_object_copies":215746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":107873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81475,"num_write_kb":325813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":8315497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108000,"ondisk_log_size":108000,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1423316,"kb_used_data":867452,"kb_used_omap":12,"kb_used_meta":555827,"kb_avail":281692204,"statfs":{"total":289910292480,"available":288452816896,"internally_reserved":0,"allocated":888270848,"data_stored":887604685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":569166958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,7,10,12,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32538624,"num_objects":7944,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-410,"num_objects_unfound":0,"num_objects_dirty":7944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":31776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006906"},"pg_stats":[{"pgid":"2.f","version":"35'6819","reported_seq":6880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:18.999534+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:18.999534+0000","last_peered":"2024-04-22T08:29:18.999534+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:18.999534+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6819,"log_dups_size":0,"ondisk_log_size":6819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27930624,"num_objects":6819,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":27276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3025},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6776","reported_seq":6887,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.687815+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:19.687815+0000","last_peered":"2024-04-22T08:29:19.687815+0000","last_clean":"2024-04-22T08:29:19.687815+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:19.687815+0000","last_undegraded":"2024-04-22T08:29:19.687815+0000","last_fullsized":"2024-04-22T08:29:19.687815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27729920,"num_objects":6770,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":15328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6711","reported_seq":6831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.696937+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:19.696937+0000","last_peered":"2024-04-22T08:29:19.696937+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:19.696937+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27471872,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":14152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6805","reported_seq":6833,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.721335+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:19.721335+0000","last_peered":"2024-04-22T08:29:19.721335+0000","last_clean":"2024-04-22T08:29:19.721335+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:19.721335+0000","last_undegraded":"2024-04-22T08:29:19.721335+0000","last_fullsized":"2024-04-22T08:29:19.721335+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27873280,"num_objects":6805,"num_object_clones":0,"num_object_copies":13610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":27220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6746","reported_seq":6881,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.033774+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:19.033774+0000","last_peered":"2024-04-22T08:29:19.033774+0000","last_clean":"2024-04-22T08:29:19.033774+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:19.033774+0000","last_undegraded":"2024-04-22T08:29:19.033774+0000","last_fullsized":"2024-04-22T08:29:19.033774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27602944,"num_objects":6739,"num_object_clones":0,"num_object_copies":13478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3838,"num_write_kb":15352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6835","reported_seq":6995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:18.959143+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:18.959143+0000","last_peered":"2024-04-22T08:29:18.959143+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:18.959143+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27975680,"num_objects":6830,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":14816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2951},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6670","reported_seq":10886,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:19.043411+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:19.043411+0000","last_peered":"2024-04-22T08:29:19.043411+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:19.043411+0000","last_undegraded":"2024-04-22T08:29:19.043411+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27295744,"num_objects":6664,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":6664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3038,"num_write_kb":12152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2028,"num_bytes_recovered":8298496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4909},{"shards":"0,2","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6651","reported_seq":6791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.048851+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:19.048851+0000","last_peered":"2024-04-22T08:29:19.048851+0000","last_clean":"2024-04-22T08:29:19.048851+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:19.048851+0000","last_undegraded":"2024-04-22T08:29:19.048851+0000","last_fullsized":"2024-04-22T08:29:19.048851+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27234304,"num_objects":6649,"num_object_clones":0,"num_object_copies":13298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3299,"num_write_kb":13196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6886","reported_seq":7006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673724+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:19.673724+0000","last_peered":"2024-04-22T08:29:19.673724+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:19.673724+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27940050,"num_objects":6823,"num_object_clones":0,"num_object_copies":13646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":27288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6743","reported_seq":6818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.028119+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:19.028119+0000","last_peered":"2024-04-22T08:29:19.028119+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:19.028119+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6743,"log_dups_size":0,"ondisk_log_size":6743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587794,"num_objects":6737,"num_object_clones":0,"num_object_copies":13474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6741,"num_write_kb":26944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2966},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6723","reported_seq":6867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673553+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:19.673553+0000","last_peered":"2024-04-22T08:29:19.673553+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:19.673553+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6732","reported_seq":6785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.691998+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:19.691998+0000","last_peered":"2024-04-22T08:29:19.691998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:19.691998+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27550930,"num_objects":6728,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":26907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2925}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6696","reported_seq":6773,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.067677+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:19.067677+0000","last_peered":"2024-04-22T08:29:19.067677+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:19.067677+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27419241,"num_objects":6695,"num_object_clones":0,"num_object_copies":13390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6695,"num_write_kb":26777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2974},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6667","reported_seq":6838,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.036962+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:19.036962+0000","last_peered":"2024-04-22T08:29:19.036962+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:19.036962+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2898,"num_write_kb":11592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2822},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6738","reported_seq":6812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.001594+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:19.001594+0000","last_peered":"2024-04-22T08:29:19.001594+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:19.001594+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27583698,"num_objects":6736,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6736,"num_write_kb":26938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3031},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6802","reported_seq":6943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.015151+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:19.015151+0000","last_peered":"2024-04-22T08:29:19.015151+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:19.015151+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6802,"log_dups_size":0,"ondisk_log_size":6802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27814308,"num_objects":6794,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":27167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2913},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":441802581,"num_objects":107873,"num_object_clones":0,"num_object_copies":215746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":107873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81475,"num_write_kb":325813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":8315497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":889106432,"data_stored":889026415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108000,"ondisk_log_size":108000,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117024,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509112,"kb_used_data":337200,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93862728,"statfs":{"total":96636764160,"available":96115433472,"internally_reserved":0,"allocated":345292800,"data_stored":345091785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.8340000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920786,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304644,"kb_used_data":146684,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94067196,"statfs":{"total":96636764160,"available":96324808704,"internally_reserved":0,"allocated":150204416,"data_stored":149933821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":161738444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":2.601}]}]},{"osd":0,"up_from":21,"seq":90194313251,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609560,"kb_used_data":383568,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93762280,"statfs":{"total":96636764160,"available":96012574720,"internally_reserved":0,"allocated":392773632,"data_stored":392579079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":231403219},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,7,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.372},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393416704,"data_stored":393413225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":150188032,"data_stored":150114973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345501696,"data_stored":345498217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:21.853 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 129 20 109170 109150 3.30412 2.09375 0.0502585 0.0236367 2024-04-22T08:29:22.856 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 130 20 109710 109690 3.29485 2.10938 0.0644147 0.0237037 2024-04-22T08:29:23.858 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 131 20 110239 110219 3.28542 2.06641 0.0382623 0.0237723 2024-04-22T08:29:24.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:24.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 132 20 110731 110711 3.27506 1.92188 0.0335641 0.0238421 2024-04-22T08:29:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:24.983 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:25.024 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-04-22T08:29:24.185230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":448487253,"num_objects":109505,"num_object_clones":0,"num_object_copies":219010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":109505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83107,"num_write_kb":332341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109632,"ondisk_log_size":109632,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491912,"kb_used_data":883568,"kb_used_omap":12,"kb_used_meta":608307,"kb_avail":281623608,"statfs":{"total":289910292480,"available":288382574592,"internally_reserved":0,"allocated":904773632,"data_stored":904107469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":622906478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,7,6,2,9,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28340224,"num_objects":6919,"num_object_clones":0,"num_object_copies":13838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-423,"num_objects_unfound":0,"num_objects_dirty":6919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6919,"num_write_kb":27676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":1732608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005315"},"pg_stats":[{"pgid":"2.f","version":"35'6977","reported_seq":7038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.052681+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:24.052681+0000","last_peered":"2024-04-22T08:29:24.052681+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:24.052681+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3183},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6776","reported_seq":6887,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.687815+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:19.687815+0000","last_peered":"2024-04-22T08:29:19.687815+0000","last_clean":"2024-04-22T08:29:19.687815+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:19.687815+0000","last_undegraded":"2024-04-22T08:29:19.687815+0000","last_fullsized":"2024-04-22T08:29:19.687815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27729920,"num_objects":6770,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":15328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6711","reported_seq":6831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.696937+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:19.696937+0000","last_peered":"2024-04-22T08:29:19.696937+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:19.696937+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27471872,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":14152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6805","reported_seq":6833,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.721335+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:19.721335+0000","last_peered":"2024-04-22T08:29:19.721335+0000","last_clean":"2024-04-22T08:29:19.721335+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:19.721335+0000","last_undegraded":"2024-04-22T08:29:19.721335+0000","last_fullsized":"2024-04-22T08:29:19.721335+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27873280,"num_objects":6805,"num_object_clones":0,"num_object_copies":13610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":27220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6921","reported_seq":7056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.034153+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:24.034153+0000","last_peered":"2024-04-22T08:29:24.034153+0000","last_clean":"2024-04-22T08:29:24.034153+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:24.034153+0000","last_undegraded":"2024-04-22T08:29:24.034153+0000","last_fullsized":"2024-04-22T08:29:24.034153+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6921,"log_dups_size":0,"ondisk_log_size":6921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28319744,"num_objects":6914,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":16052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6982","reported_seq":7142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067673+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:24.067673+0000","last_peered":"2024-04-22T08:29:24.067673+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:24.067673+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":15404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3098},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6821","reported_seq":11335,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:24.053833+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:24.053833+0000","last_peered":"2024-04-22T08:29:24.053833+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:24.053833+0000","last_undegraded":"2024-04-22T08:29:24.053833+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27914240,"num_objects":6815,"num_object_clones":0,"num_object_copies":13630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":8908800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5209},{"shards":"0,2","objects":1606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6812","reported_seq":6954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.011261+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:24.011261+0000","last_peered":"2024-04-22T08:29:24.011261+0000","last_clean":"2024-04-22T08:29:24.011261+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:24.011261+0000","last_undegraded":"2024-04-22T08:29:24.011261+0000","last_fullsized":"2024-04-22T08:29:24.011261+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27893760,"num_objects":6810,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":13840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6886","reported_seq":7006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673724+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:19.673724+0000","last_peered":"2024-04-22T08:29:19.673724+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:19.673724+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27940050,"num_objects":6823,"num_object_clones":0,"num_object_copies":13646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":27288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6918","reported_seq":6989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067282+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:24.067282+0000","last_peered":"2024-04-22T08:29:24.067282+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:24.067282+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28304594,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6723","reported_seq":6867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673553+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:19.673553+0000","last_peered":"2024-04-22T08:29:19.673553+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:19.673553+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6732","reported_seq":6785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.691998+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:19.691998+0000","last_peered":"2024-04-22T08:29:19.691998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:19.691998+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27550930,"num_objects":6728,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":26907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2925}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6864","reported_seq":6941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.010893+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:24.010893+0000","last_peered":"2024-04-22T08:29:24.010893+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:24.010893+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107369,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":27449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3142},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6812","reported_seq":6984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.032006+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:24.032006+0000","last_peered":"2024-04-22T08:29:24.032006+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:24.032006+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27869184,"num_objects":6804,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":12172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2967},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6912","reported_seq":6986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.054104+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:24.054104+0000","last_peered":"2024-04-22T08:29:24.054104+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:24.054104+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28296402,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":27634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3205},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6980","reported_seq":7121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.004325+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:24.004325+0000","last_peered":"2024-04-22T08:29:24.004325+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:24.004325+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28543396,"num_objects":6972,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3091},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":448487253,"num_objects":109505,"num_object_clones":0,"num_object_copies":219010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":109505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83107,"num_write_kb":332341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905572352,"data_stored":905492335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109632,"ondisk_log_size":109632,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117024,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509112,"kb_used_data":337200,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93862728,"statfs":{"total":96636764160,"available":96115433472,"internally_reserved":0,"allocated":345292800,"data_stored":345091785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.8340000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321036,"kb_used_data":154884,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":94050804,"statfs":{"total":96636764160,"available":96308023296,"internally_reserved":0,"allocated":158601216,"data_stored":158330621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":170127052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.597},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":1.7350000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313252,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661764,"kb_used_data":391484,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93710076,"statfs":{"total":96636764160,"available":95959117824,"internally_reserved":0,"allocated":400879616,"data_stored":400685063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.251},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.0429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401342464,"data_stored":401338985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158728192,"data_stored":158655133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345501696,"data_stored":345498217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:25.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:25.856 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:25.863 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 133 20 111212 111192 3.26456 1.87891 0.0212082 0.0239223 2024-04-22T08:29:25.923 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-04-22T08:29:24.185230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":448487253,"num_objects":109505,"num_object_clones":0,"num_object_copies":219010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":109505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83107,"num_write_kb":332341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109632,"ondisk_log_size":109632,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491912,"kb_used_data":883568,"kb_used_omap":12,"kb_used_meta":608307,"kb_avail":281623608,"statfs":{"total":289910292480,"available":288382574592,"internally_reserved":0,"allocated":904773632,"data_stored":904107469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":622906478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,7,6,2,9,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28340224,"num_objects":6919,"num_object_clones":0,"num_object_copies":13838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-423,"num_objects_unfound":0,"num_objects_dirty":6919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6919,"num_write_kb":27676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":1732608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005315"},"pg_stats":[{"pgid":"2.f","version":"35'6977","reported_seq":7038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.052681+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:24.052681+0000","last_peered":"2024-04-22T08:29:24.052681+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:24.052681+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3183},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6776","reported_seq":6887,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.687815+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:19.687815+0000","last_peered":"2024-04-22T08:29:19.687815+0000","last_clean":"2024-04-22T08:29:19.687815+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:19.687815+0000","last_undegraded":"2024-04-22T08:29:19.687815+0000","last_fullsized":"2024-04-22T08:29:19.687815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27729920,"num_objects":6770,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":15328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6711","reported_seq":6831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.696937+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:19.696937+0000","last_peered":"2024-04-22T08:29:19.696937+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:19.696937+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27471872,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":14152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":2952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6805","reported_seq":6833,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:19.721335+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:19.721335+0000","last_peered":"2024-04-22T08:29:19.721335+0000","last_clean":"2024-04-22T08:29:19.721335+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:19.721335+0000","last_undegraded":"2024-04-22T08:29:19.721335+0000","last_fullsized":"2024-04-22T08:29:19.721335+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27873280,"num_objects":6805,"num_object_clones":0,"num_object_copies":13610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":27220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6921","reported_seq":7056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.034153+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:24.034153+0000","last_peered":"2024-04-22T08:29:24.034153+0000","last_clean":"2024-04-22T08:29:24.034153+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:24.034153+0000","last_undegraded":"2024-04-22T08:29:24.034153+0000","last_fullsized":"2024-04-22T08:29:24.034153+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6921,"log_dups_size":0,"ondisk_log_size":6921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28319744,"num_objects":6914,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":16052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6982","reported_seq":7142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067673+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:24.067673+0000","last_peered":"2024-04-22T08:29:24.067673+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:24.067673+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":15404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3098},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6821","reported_seq":11335,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:24.053833+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:24.053833+0000","last_peered":"2024-04-22T08:29:24.053833+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:24.053833+0000","last_undegraded":"2024-04-22T08:29:24.053833+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27914240,"num_objects":6815,"num_object_clones":0,"num_object_copies":13630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":8908800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5209},{"shards":"0,2","objects":1606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6812","reported_seq":6954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.011261+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:24.011261+0000","last_peered":"2024-04-22T08:29:24.011261+0000","last_clean":"2024-04-22T08:29:24.011261+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:24.011261+0000","last_undegraded":"2024-04-22T08:29:24.011261+0000","last_fullsized":"2024-04-22T08:29:24.011261+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27893760,"num_objects":6810,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":13840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'6886","reported_seq":7006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673724+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:19.673724+0000","last_peered":"2024-04-22T08:29:19.673724+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:19.673724+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27940050,"num_objects":6823,"num_object_clones":0,"num_object_copies":13646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":27288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6918","reported_seq":6989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067282+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:24.067282+0000","last_peered":"2024-04-22T08:29:24.067282+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:24.067282+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28304594,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6723","reported_seq":6867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.673553+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:19.673553+0000","last_peered":"2024-04-22T08:29:19.673553+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:19.673553+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27516928,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":2968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6732","reported_seq":6785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:19.691998+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:19.691998+0000","last_peered":"2024-04-22T08:29:19.691998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:19.691998+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27550930,"num_objects":6728,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":26907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":2925}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6864","reported_seq":6941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.010893+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:24.010893+0000","last_peered":"2024-04-22T08:29:24.010893+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:24.010893+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107369,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":27449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3142},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6812","reported_seq":6984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.032006+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:24.032006+0000","last_peered":"2024-04-22T08:29:24.032006+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:24.032006+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27869184,"num_objects":6804,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":12172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2967},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6912","reported_seq":6986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.054104+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:24.054104+0000","last_peered":"2024-04-22T08:29:24.054104+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:24.054104+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28296402,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":27634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3205},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6980","reported_seq":7121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.004325+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:24.004325+0000","last_peered":"2024-04-22T08:29:24.004325+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:24.004325+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28543396,"num_objects":6972,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3091},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":448487253,"num_objects":109505,"num_object_clones":0,"num_object_copies":219010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":109505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83107,"num_write_kb":332341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905572352,"data_stored":905492335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109632,"ondisk_log_size":109632,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117024,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509112,"kb_used_data":337200,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93862728,"statfs":{"total":96636764160,"available":96115433472,"internally_reserved":0,"allocated":345292800,"data_stored":345091785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.8340000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321036,"kb_used_data":154884,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":94050804,"statfs":{"total":96636764160,"available":96308023296,"internally_reserved":0,"allocated":158601216,"data_stored":158330621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":170127052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.597},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":1.7350000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313252,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661764,"kb_used_data":391484,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93710076,"statfs":{"total":96636764160,"available":95959117824,"internally_reserved":0,"allocated":400879616,"data_stored":400685063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.251},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.0429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401342464,"data_stored":401338985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158728192,"data_stored":158655133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345501696,"data_stored":345498217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:25.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:25.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'6977", 'reported_seq': 7038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.052681+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:24.052681+0000', 'last_peered': '2024-04-22T08:29:24.052681+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:24.052681+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6977, 'log_dups_size': 0, 'ondisk_log_size': 6977, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28577792, 'num_objects': 6977, 'num_object_clones': 0, 'num_object_copies': 13954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6977, 'num_write_kb': 27908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3183}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'6711", 'reported_seq': 6831, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:19.696937+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:19.696937+0000', 'last_peered': '2024-04-22T08:29:19.696937+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:19.696937+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 6711, 'log_dups_size': 0, 'ondisk_log_size': 6711, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27471872, 'num_objects': 6707, 'num_object_clones': 0, 'num_object_copies': 13414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3538, 'num_write_kb': 14152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 2952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'6982", 'reported_seq': 7142, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.067673+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:24.067673+0000', 'last_peered': '2024-04-22T08:29:24.067673+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:24.067673+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 6982, 'log_dups_size': 0, 'ondisk_log_size': 6982, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28577792, 'num_objects': 6977, 'num_object_clones': 0, 'num_object_copies': 13954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3851, 'num_write_kb': 15404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3098}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'6821", 'reported_seq': 11335, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:24.053833+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:24.053833+0000', 'last_peered': '2024-04-22T08:29:24.053833+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:24.053833+0000', 'last_undegraded': '2024-04-22T08:29:24.053833+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 6821, 'log_dups_size': 0, 'ondisk_log_size': 6821, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27914240, 'num_objects': 6815, 'num_object_clones': 0, 'num_object_copies': 13630, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1606, 'num_objects_unfound': 0, 'num_objects_dirty': 6815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3189, 'num_write_kb': 12756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2177, 'num_bytes_recovered': 8908800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5209}, {'shards': '0,2', 'objects': 1606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'6886", 'reported_seq': 7006, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:19.673724+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:19.673724+0000', 'last_peered': '2024-04-22T08:29:19.673724+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:19.673724+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 6886, 'log_dups_size': 0, 'ondisk_log_size': 6886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27940050, 'num_objects': 6823, 'num_object_clones': 0, 'num_object_copies': 13646, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6826, 'num_write_kb': 27288, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 2935}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:25.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'6918", 'reported_seq': 6989, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.067282+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:24.067282+0000', 'last_peered': '2024-04-22T08:29:24.067282+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:24.067282+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6918, 'log_dups_size': 0, 'ondisk_log_size': 6918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28304594, 'num_objects': 6912, 'num_object_clones': 0, 'num_object_copies': 13824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6916, 'num_write_kb': 27644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3141}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'6723", 'reported_seq': 6867, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:19.673553+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:19.673553+0000', 'last_peered': '2024-04-22T08:29:19.673553+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:19.673553+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 6723, 'log_dups_size': 0, 'ondisk_log_size': 6723, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27516928, 'num_objects': 6718, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3177, 'num_write_kb': 12708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 2968}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'6732", 'reported_seq': 6785, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:19.691998+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:19.691998+0000', 'last_peered': '2024-04-22T08:29:19.691998+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:19.691998+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6732, 'log_dups_size': 0, 'ondisk_log_size': 6732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27550930, 'num_objects': 6728, 'num_object_clones': 0, 'num_object_copies': 13456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6730, 'num_write_kb': 26907, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 2925}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'6864", 'reported_seq': 6941, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.010893+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:24.010893+0000', 'last_peered': '2024-04-22T08:29:24.010893+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:24.010893+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6864, 'log_dups_size': 0, 'ondisk_log_size': 6864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28107369, 'num_objects': 6863, 'num_object_clones': 0, 'num_object_copies': 13726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6863, 'num_write_kb': 27449, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3142}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'6812", 'reported_seq': 6984, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.032006+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:24.032006+0000', 'last_peered': '2024-04-22T08:29:24.032006+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:24.032006+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 6812, 'log_dups_size': 0, 'ondisk_log_size': 6812, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27869184, 'num_objects': 6804, 'num_object_clones': 0, 'num_object_copies': 13608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3043, 'num_write_kb': 12172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2967}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:25.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'6912", 'reported_seq': 6986, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.054104+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:24.054104+0000', 'last_peered': '2024-04-22T08:29:24.054104+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:24.054104+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 6912, 'log_dups_size': 0, 'ondisk_log_size': 6912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28296402, 'num_objects': 6910, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6910, 'num_write_kb': 27634, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3205}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.927 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:25.927 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'6980", 'reported_seq': 7121, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:24.004325+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:24.004325+0000', 'last_peered': '2024-04-22T08:29:24.004325+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:24.004325+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 6980, 'log_dups_size': 0, 'ondisk_log_size': 6980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28543396, 'num_objects': 6972, 'num_object_clones': 0, 'num_object_copies': 13944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6977, 'num_write_kb': 27879, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3091}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:25.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:26.716 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:26.717 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/220980 objects degraded (18.918%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":201,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":110490,"data_bytes":452521813,"bytes_used":1541484544,"bytes_avail":288368807936,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":220980,"degraded_ratio":0.18917549099466016,"misplaced_objects":1606,"misplaced_total":220980,"misplaced_ratio":0.0072676260295049325,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":98942,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":2129659,"read_op_per_sec":0,"write_op_per_sec":519},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:26.780 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:26.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:26.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 134 20 111692 111672 3.25419 1.875 0.0853091 0.0239983 2024-04-22T08:29:27.444 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:27.445 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:27.509 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-22T08:29:26.185738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452521813,"num_objects":110490,"num_object_clones":0,"num_object_copies":220980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":110490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84092,"num_write_kb":336281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110617,"ondisk_log_size":110617,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1505356,"kb_used_data":888820,"kb_used_omap":12,"kb_used_meta":616499,"kb_avail":281610164,"statfs":{"total":289910292480,"available":288368807936,"internally_reserved":0,"allocated":910151680,"data_stored":909485517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":631295086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,7,6,4,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25567232,"num_objects":6242,"num_object_clones":0,"num_object_copies":12484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-290,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":24968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005315"},"pg_stats":[{"pgid":"2.f","version":"35'6977","reported_seq":7038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.052681+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:24.052681+0000","last_peered":"2024-04-22T08:29:24.052681+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:24.052681+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3183},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'6944","reported_seq":7056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.664614+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:24.664614+0000","last_peered":"2024-04-22T08:29:24.664614+0000","last_clean":"2024-04-22T08:29:24.664614+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:24.664614+0000","last_undegraded":"2024-04-22T08:29:24.664614+0000","last_fullsized":"2024-04-22T08:29:24.664614+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":6944,"log_dups_size":0,"ondisk_log_size":6944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28418048,"num_objects":6938,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4000,"num_write_kb":16000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'6873","reported_seq":6992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.683553+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:24.683553+0000","last_peered":"2024-04-22T08:29:24.683553+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:24.683553+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28135424,"num_objects":6869,"num_object_clones":0,"num_object_copies":13738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":14800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'6943","reported_seq":6971,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.731654+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:24.731654+0000","last_peered":"2024-04-22T08:29:24.731654+0000","last_clean":"2024-04-22T08:29:24.731654+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:24.731654+0000","last_undegraded":"2024-04-22T08:29:24.731654+0000","last_fullsized":"2024-04-22T08:29:24.731654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28438528,"num_objects":6943,"num_object_clones":0,"num_object_copies":13886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":27772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'6921","reported_seq":7056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.034153+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:24.034153+0000","last_peered":"2024-04-22T08:29:24.034153+0000","last_clean":"2024-04-22T08:29:24.034153+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:24.034153+0000","last_undegraded":"2024-04-22T08:29:24.034153+0000","last_fullsized":"2024-04-22T08:29:24.034153+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":6921,"log_dups_size":0,"ondisk_log_size":6921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28319744,"num_objects":6914,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":16052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'6982","reported_seq":7142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067673+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:24.067673+0000","last_peered":"2024-04-22T08:29:24.067673+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:24.067673+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":15404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3098},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6821","reported_seq":11335,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:24.053833+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:24.053833+0000","last_peered":"2024-04-22T08:29:24.053833+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:24.053833+0000","last_undegraded":"2024-04-22T08:29:24.053833+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27914240,"num_objects":6815,"num_object_clones":0,"num_object_copies":13630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":8908800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5209},{"shards":"0,2","objects":1606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6812","reported_seq":6954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:24.011261+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:24.011261+0000","last_peered":"2024-04-22T08:29:24.011261+0000","last_clean":"2024-04-22T08:29:24.011261+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:24.011261+0000","last_undegraded":"2024-04-22T08:29:24.011261+0000","last_fullsized":"2024-04-22T08:29:24.011261+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27893760,"num_objects":6810,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":13840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7053","reported_seq":7173,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.717601+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:24.717601+0000","last_peered":"2024-04-22T08:29:24.717601+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:24.717601+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28624082,"num_objects":6990,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6993,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'6918","reported_seq":6989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.067282+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:24.067282+0000","last_peered":"2024-04-22T08:29:24.067282+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:24.067282+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28304594,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'6887","reported_seq":7031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.728065+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:24.728065+0000","last_peered":"2024-04-22T08:29:24.728065+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:24.728065+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28188672,"num_objects":6882,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":13364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'6918","reported_seq":6971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.632495+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:24.632495+0000","last_peered":"2024-04-22T08:29:24.632495+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:24.632495+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28312786,"num_objects":6914,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":27651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'6864","reported_seq":6941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.010893+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:24.010893+0000","last_peered":"2024-04-22T08:29:24.010893+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:24.010893+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107369,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":27449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3142},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6812","reported_seq":6984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.032006+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:24.032006+0000","last_peered":"2024-04-22T08:29:24.032006+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:24.032006+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27869184,"num_objects":6804,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":12172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2967},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'6912","reported_seq":6986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.054104+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:24.054104+0000","last_peered":"2024-04-22T08:29:24.054104+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:24.054104+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28296402,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":27634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3205},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'6980","reported_seq":7121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:24.004325+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:24.004325+0000","last_peered":"2024-04-22T08:29:24.004325+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:24.004325+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28543396,"num_objects":6972,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":27879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3091},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":452521813,"num_objects":110490,"num_object_clones":0,"num_object_copies":220980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":110490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84092,"num_write_kb":336281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":8925801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":910962688,"data_stored":910882671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110617,"ondisk_log_size":110617,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117025,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522556,"kb_used_data":342452,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93849284,"statfs":{"total":96636764160,"available":96101666816,"internally_reserved":0,"allocated":350670848,"data_stored":350469833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184413903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.4390000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321036,"kb_used_data":154884,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":94050804,"statfs":{"total":96636764160,"available":96308023296,"internally_reserved":0,"allocated":158601216,"data_stored":158330621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":170127052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.597},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":1.7350000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313252,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661764,"kb_used_data":391484,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93710076,"statfs":{"total":96636764160,"available":95959117824,"internally_reserved":0,"allocated":400879616,"data_stored":400685063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.251},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.0429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401342464,"data_stored":401338985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158728192,"data_stored":158655133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":350892032,"data_stored":350888553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:27.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 135 20 112172 112152 3.24398 1.875 0.0165333 0.0240741 2024-04-22T08:29:28.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 136 20 112688 112668 3.23495 2.01562 0.00996324 0.0241418 2024-04-22T08:29:29.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 137 20 113202 113182 3.22599 2.00781 0.00810294 0.0242091 2024-04-22T08:29:30.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:30.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 138 20 113717 113697 3.21719 2.01172 0.0227088 0.0242761 2024-04-22T08:29:31.428 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:31.429 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-04-22T08:29:30.187479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462520149,"num_objects":112931,"num_object_clones":0,"num_object_copies":225862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":112931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86533,"num_write_kb":346045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":9548393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113058,"ondisk_log_size":113058,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1543384,"kb_used_data":908800,"kb_used_omap":12,"kb_used_meta":634547,"kb_avail":281572136,"statfs":{"total":289910292480,"available":288329867264,"internally_reserved":0,"allocated":930611200,"data_stored":929945037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":649776238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,3,3,8,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31612928,"num_objects":7718,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-442,"num_objects_unfound":0,"num_objects_dirty":7718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7718,"num_write_kb":30872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"2.f","version":"35'7127","reported_seq":7188,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.995848+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:28.995848+0000","last_peered":"2024-04-22T08:29:28.995848+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:28.995848+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29192192,"num_objects":7127,"num_object_clones":0,"num_object_copies":14254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":28508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3333},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7083","reported_seq":7195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.748424+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:29.748424+0000","last_peered":"2024-04-22T08:29:29.748424+0000","last_clean":"2024-04-22T08:29:29.748424+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:29.748424+0000","last_undegraded":"2024-04-22T08:29:29.748424+0000","last_fullsized":"2024-04-22T08:29:29.748424+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":16556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7045","reported_seq":7166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.755451+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:29.755451+0000","last_peered":"2024-04-22T08:29:29.755451+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:29.755451+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28839936,"num_objects":7041,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7090","reported_seq":7118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.662181+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:29.662181+0000","last_peered":"2024-04-22T08:29:29.662181+0000","last_clean":"2024-04-22T08:29:29.662181+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:29.662181+0000","last_undegraded":"2024-04-22T08:29:29.662181+0000","last_fullsized":"2024-04-22T08:29:29.662181+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7090,"log_dups_size":0,"ondisk_log_size":7090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29040640,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":28360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7060","reported_seq":7196,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.080697+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:29.080697+0000","last_peered":"2024-04-22T08:29:29.080697+0000","last_clean":"2024-04-22T08:29:29.080697+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:29.080697+0000","last_undegraded":"2024-04-22T08:29:29.080697+0000","last_fullsized":"2024-04-22T08:29:29.080697+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28889088,"num_objects":7053,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":16608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7136","reported_seq":7296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.057549+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:29.057549+0000","last_peered":"2024-04-22T08:29:29.057549+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:29.057549+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29208576,"num_objects":7131,"num_object_clones":0,"num_object_copies":14262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":16020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3252},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6964","reported_seq":11782,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:29.092030+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:29.092030+0000","last_peered":"2024-04-22T08:29:29.092030+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:29.092030+0000","last_undegraded":"2024-04-22T08:29:29.092030+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28499968,"num_objects":6958,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":6958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":13328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2329,"num_bytes_recovered":9531392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5504},{"shards":"0,2","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6971","reported_seq":7113,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.044507+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:29.044507+0000","last_peered":"2024-04-22T08:29:29.044507+0000","last_clean":"2024-04-22T08:29:29.044507+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:29.044507+0000","last_undegraded":"2024-04-22T08:29:29.044507+0000","last_fullsized":"2024-04-22T08:29:29.044507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6971,"log_dups_size":0,"ondisk_log_size":6971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28545024,"num_objects":6969,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3619,"num_write_kb":14476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7210","reported_seq":7330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.721385+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:29.721385+0000","last_peered":"2024-04-22T08:29:29.721385+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:29.721385+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267154,"num_objects":7147,"num_object_clones":0,"num_object_copies":14294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":28584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7067","reported_seq":7142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.969952+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:28.969952+0000","last_peered":"2024-04-22T08:29:28.969952+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:28.969952+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28914898,"num_objects":7061,"num_object_clones":0,"num_object_copies":14122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":28240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3290},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7035","reported_seq":7179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.708524+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:29.708524+0000","last_peered":"2024-04-22T08:29:29.708524+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:29.708524+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28794880,"num_objects":7030,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":13956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3280}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7060","reported_seq":7113,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.602810+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:29.602810+0000","last_peered":"2024-04-22T08:29:29.602810+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:29.602810+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28894418,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":28219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7034","reported_seq":7105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.005960+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:29.005960+0000","last_peered":"2024-04-22T08:29:29.005960+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:29.005960+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28803689,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7033,"num_write_kb":28129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3312},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6981","reported_seq":7153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.055369+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:29.055369+0000","last_peered":"2024-04-22T08:29:29.055369+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:29.055369+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6981,"log_dups_size":0,"ondisk_log_size":6981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28561408,"num_objects":6973,"num_object_clones":0,"num_object_copies":13946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":12848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3136},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7061","reported_seq":7135,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.940008+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:28.940008+0000","last_peered":"2024-04-22T08:29:28.940008+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:28.940008+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28906706,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":28230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3354},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7134","reported_seq":7275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.060673+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:29.060673+0000","last_peered":"2024-04-22T08:29:29.060673+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:29.060673+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29174180,"num_objects":7126,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":28495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3245},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":462520149,"num_objects":112931,"num_object_clones":0,"num_object_copies":225862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":112931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86533,"num_write_kb":346045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":9548393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931762176,"data_stored":931682159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113058,"ondisk_log_size":113058,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531412,"kb_used_data":347212,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93840428,"statfs":{"total":96636764160,"available":96092598272,"internally_reserved":0,"allocated":355545088,"data_stored":355344073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.4390000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920788,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338876,"kb_used_data":162868,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":94032964,"statfs":{"total":96636764160,"available":96289755136,"internally_reserved":0,"allocated":166776832,"data_stored":166506237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":180219596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.28899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313254,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673096,"kb_used_data":398720,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93698744,"statfs":{"total":96636764160,"available":95947513856,"internally_reserved":0,"allocated":408289280,"data_stored":408094727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":280948435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.251},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.0429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408858624,"data_stored":408855145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167067648,"data_stored":166994589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355835904,"data_stored":355832425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:31.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:31.860 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 139 20 114193 114173 3.20743 1.85938 0.100205 0.0243479 2024-04-22T08:29:32.380 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:32.381 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:32.449 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-04-22T08:29:32.188008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462520149,"num_objects":112931,"num_object_clones":0,"num_object_copies":225862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":112931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86533,"num_write_kb":346045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":9548393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113058,"ondisk_log_size":113058,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1543384,"kb_used_data":908800,"kb_used_omap":12,"kb_used_meta":634547,"kb_avail":281572136,"statfs":{"total":289910292480,"available":288329867264,"internally_reserved":0,"allocated":930611200,"data_stored":929945037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":649776238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,3,3,8,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20717568,"num_objects":5058,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-301,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5058,"num_write_kb":20232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":1232896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003853"},"pg_stats":[{"pgid":"2.f","version":"35'7127","reported_seq":7188,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.995848+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:28.995848+0000","last_peered":"2024-04-22T08:29:28.995848+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:28.995848+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29192192,"num_objects":7127,"num_object_clones":0,"num_object_copies":14254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":28508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3333},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7083","reported_seq":7195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.748424+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:29.748424+0000","last_peered":"2024-04-22T08:29:29.748424+0000","last_clean":"2024-04-22T08:29:29.748424+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:29.748424+0000","last_undegraded":"2024-04-22T08:29:29.748424+0000","last_fullsized":"2024-04-22T08:29:29.748424+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":16556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7045","reported_seq":7166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.755451+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:29.755451+0000","last_peered":"2024-04-22T08:29:29.755451+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:29.755451+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28839936,"num_objects":7041,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7090","reported_seq":7118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.662181+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:29.662181+0000","last_peered":"2024-04-22T08:29:29.662181+0000","last_clean":"2024-04-22T08:29:29.662181+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:29.662181+0000","last_undegraded":"2024-04-22T08:29:29.662181+0000","last_fullsized":"2024-04-22T08:29:29.662181+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7090,"log_dups_size":0,"ondisk_log_size":7090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29040640,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":28360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7060","reported_seq":7196,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.080697+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:29.080697+0000","last_peered":"2024-04-22T08:29:29.080697+0000","last_clean":"2024-04-22T08:29:29.080697+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:29.080697+0000","last_undegraded":"2024-04-22T08:29:29.080697+0000","last_fullsized":"2024-04-22T08:29:29.080697+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28889088,"num_objects":7053,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":16608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7136","reported_seq":7296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.057549+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:29.057549+0000","last_peered":"2024-04-22T08:29:29.057549+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:29.057549+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29208576,"num_objects":7131,"num_object_clones":0,"num_object_copies":14262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":16020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3252},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'6964","reported_seq":11782,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:29.092030+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:29.092030+0000","last_peered":"2024-04-22T08:29:29.092030+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:29.092030+0000","last_undegraded":"2024-04-22T08:29:29.092030+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28499968,"num_objects":6958,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":6958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":13328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2329,"num_bytes_recovered":9531392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5504},{"shards":"0,2","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'6971","reported_seq":7113,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.044507+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:29.044507+0000","last_peered":"2024-04-22T08:29:29.044507+0000","last_clean":"2024-04-22T08:29:29.044507+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:29.044507+0000","last_undegraded":"2024-04-22T08:29:29.044507+0000","last_fullsized":"2024-04-22T08:29:29.044507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":6971,"log_dups_size":0,"ondisk_log_size":6971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28545024,"num_objects":6969,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3619,"num_write_kb":14476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7210","reported_seq":7330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.721385+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:29.721385+0000","last_peered":"2024-04-22T08:29:29.721385+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:29.721385+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267154,"num_objects":7147,"num_object_clones":0,"num_object_copies":14294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":28584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7067","reported_seq":7142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.969952+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:28.969952+0000","last_peered":"2024-04-22T08:29:28.969952+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:28.969952+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28914898,"num_objects":7061,"num_object_clones":0,"num_object_copies":14122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":28240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3290},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7035","reported_seq":7179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.708524+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:29.708524+0000","last_peered":"2024-04-22T08:29:29.708524+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:29.708524+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28794880,"num_objects":7030,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":13956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3280}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7060","reported_seq":7113,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.602810+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:29.602810+0000","last_peered":"2024-04-22T08:29:29.602810+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:29.602810+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28894418,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":28219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7034","reported_seq":7105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.005960+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:29.005960+0000","last_peered":"2024-04-22T08:29:29.005960+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:29.005960+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28803689,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7033,"num_write_kb":28129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3312},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'6981","reported_seq":7153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.055369+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:29.055369+0000","last_peered":"2024-04-22T08:29:29.055369+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:29.055369+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":6981,"log_dups_size":0,"ondisk_log_size":6981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28561408,"num_objects":6973,"num_object_clones":0,"num_object_copies":13946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":12848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3136},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7061","reported_seq":7135,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:28.940008+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:28.940008+0000","last_peered":"2024-04-22T08:29:28.940008+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:28.940008+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28906706,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":28230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3354},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7134","reported_seq":7275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.060673+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:29.060673+0000","last_peered":"2024-04-22T08:29:29.060673+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:29.060673+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29174180,"num_objects":7126,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":28495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3245},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":462520149,"num_objects":112931,"num_object_clones":0,"num_object_copies":225862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1454,"num_objects_unfound":0,"num_objects_dirty":112931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86533,"num_write_kb":346045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":9548393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931762176,"data_stored":931682159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113058,"ondisk_log_size":113058,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531412,"kb_used_data":347212,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93840428,"statfs":{"total":96636764160,"available":96092598272,"internally_reserved":0,"allocated":355545088,"data_stored":355344073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.4390000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920788,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338876,"kb_used_data":162868,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":94032964,"statfs":{"total":96636764160,"available":96289755136,"internally_reserved":0,"allocated":166776832,"data_stored":166506237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":180219596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.28899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313254,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673096,"kb_used_data":398720,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93698744,"statfs":{"total":96636764160,"available":95947513856,"internally_reserved":0,"allocated":408289280,"data_stored":408094727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":280948435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.251},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.0429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408858624,"data_stored":408855145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167067648,"data_stored":166994589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355835904,"data_stored":355832425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'7127", 'reported_seq': 7188, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:28.995848+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:28.995848+0000', 'last_peered': '2024-04-22T08:29:28.995848+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:28.995848+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7127, 'log_dups_size': 0, 'ondisk_log_size': 7127, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29192192, 'num_objects': 7127, 'num_object_clones': 0, 'num_object_copies': 14254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7127, 'num_write_kb': 28508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3333}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7045", 'reported_seq': 7166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.755451+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:29.755451+0000', 'last_peered': '2024-04-22T08:29:29.755451+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:29.755451+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7045, 'log_dups_size': 0, 'ondisk_log_size': 7045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28839936, 'num_objects': 7041, 'num_object_clones': 0, 'num_object_copies': 14082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3872, 'num_write_kb': 15488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 3286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:32.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'7136", 'reported_seq': 7296, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.057549+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:29.057549+0000', 'last_peered': '2024-04-22T08:29:29.057549+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:29.057549+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 7136, 'log_dups_size': 0, 'ondisk_log_size': 7136, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29208576, 'num_objects': 7131, 'num_object_clones': 0, 'num_object_copies': 14262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4005, 'num_write_kb': 16020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3252}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'6964", 'reported_seq': 11782, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:29.092030+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:29.092030+0000', 'last_peered': '2024-04-22T08:29:29.092030+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:29.092030+0000', 'last_undegraded': '2024-04-22T08:29:29.092030+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 6964, 'log_dups_size': 0, 'ondisk_log_size': 6964, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28499968, 'num_objects': 6958, 'num_object_clones': 0, 'num_object_copies': 13916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1454, 'num_objects_unfound': 0, 'num_objects_dirty': 6958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3332, 'num_write_kb': 13328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2329, 'num_bytes_recovered': 9531392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5504}, {'shards': '0,2', 'objects': 1454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'7210", 'reported_seq': 7330, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.721385+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:29.721385+0000', 'last_peered': '2024-04-22T08:29:29.721385+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:29.721385+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 7210, 'log_dups_size': 0, 'ondisk_log_size': 7210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29267154, 'num_objects': 7147, 'num_object_clones': 0, 'num_object_copies': 14294, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7150, 'num_write_kb': 28584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 3259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'7067", 'reported_seq': 7142, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:28.969952+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:28.969952+0000', 'last_peered': '2024-04-22T08:29:28.969952+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:28.969952+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7067, 'log_dups_size': 0, 'ondisk_log_size': 7067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28914898, 'num_objects': 7061, 'num_object_clones': 0, 'num_object_copies': 14122, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7065, 'num_write_kb': 28240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3290}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7035", 'reported_seq': 7179, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.708524+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:29.708524+0000', 'last_peered': '2024-04-22T08:29:29.708524+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:29.708524+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7035, 'log_dups_size': 0, 'ondisk_log_size': 7035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28794880, 'num_objects': 7030, 'num_object_clones': 0, 'num_object_copies': 14060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3489, 'num_write_kb': 13956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 3280}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:32.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7060", 'reported_seq': 7113, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.602810+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:29.602810+0000', 'last_peered': '2024-04-22T08:29:29.602810+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:29.602810+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7060, 'log_dups_size': 0, 'ondisk_log_size': 7060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28894418, 'num_objects': 7056, 'num_object_clones': 0, 'num_object_copies': 14112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7058, 'num_write_kb': 28219, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 3253}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7034", 'reported_seq': 7105, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.005960+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:29.005960+0000', 'last_peered': '2024-04-22T08:29:29.005960+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:29.005960+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7034, 'log_dups_size': 0, 'ondisk_log_size': 7034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28803689, 'num_objects': 7033, 'num_object_clones': 0, 'num_object_copies': 14066, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7033, 'num_write_kb': 28129, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3312}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'6981", 'reported_seq': 7153, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.055369+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:29.055369+0000', 'last_peered': '2024-04-22T08:29:29.055369+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:29.055369+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 6981, 'log_dups_size': 0, 'ondisk_log_size': 6981, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28561408, 'num_objects': 6973, 'num_object_clones': 0, 'num_object_copies': 13946, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3212, 'num_write_kb': 12848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3136}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'7061", 'reported_seq': 7135, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:28.940008+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:28.940008+0000', 'last_peered': '2024-04-22T08:29:28.940008+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:28.940008+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7061, 'log_dups_size': 0, 'ondisk_log_size': 7061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28906706, 'num_objects': 7059, 'num_object_clones': 0, 'num_object_copies': 14118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7059, 'num_write_kb': 28230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3354}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:32.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'7134", 'reported_seq': 7275, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:29.060673+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:29.060673+0000', 'last_peered': '2024-04-22T08:29:29.060673+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:29.060673+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 7134, 'log_dups_size': 0, 'ondisk_log_size': 7134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29174180, 'num_objects': 7126, 'num_object_clones': 0, 'num_object_copies': 14252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7131, 'num_write_kb': 28495, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3245}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:32.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:32.861 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:29:32.862865+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0244158 2024-04-22T08:29:32.861 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:29:32.861 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 140 20 114691 114671 3.19841 1.94531 0.0113318 0.0244158 2024-04-22T08:29:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:33.499 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/225862 objects degraded (18.509%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":208,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":112931,"data_bytes":462520149,"bytes_used":1580425216,"bytes_avail":288329867264,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":225862,"degraded_ratio":0.18508646872869275,"misplaced_objects":1454,"misplaced_total":225862,"misplaced_ratio":0.0064375592175753334,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":102708,"recovering_keys_per_sec":0,"num_objects_recovered":301,"num_bytes_recovered":1232896,"num_keys_recovered":0,"write_bytes_sec":1725909,"read_op_per_sec":0,"write_op_per_sec":421},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:33.552 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:33.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:33.861 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 141 20 115187 115167 3.18947 1.9375 0.0242287 0.024487 2024-04-22T08:29:34.289 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:34.291 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:34.344 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-22T08:29:34.188534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468852565,"num_objects":114477,"num_object_clones":0,"num_object_copies":228954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":114477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88079,"num_write_kb":352229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114604,"ondisk_log_size":114604,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1559008,"kb_used_data":924424,"kb_used_omap":12,"kb_used_meta":634547,"kb_avail":281556512,"statfs":{"total":289910292480,"available":288313868288,"internally_reserved":0,"allocated":946610176,"data_stored":945944013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":649776238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,2,7,7,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-454,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":1859584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003796"},"pg_stats":[{"pgid":"2.f","version":"35'7283","reported_seq":7344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.000525+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:34.000525+0000","last_peered":"2024-04-22T08:29:34.000525+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:34.000525+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29831168,"num_objects":7283,"num_object_clones":0,"num_object_copies":14566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":29132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3489},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7083","reported_seq":7195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.748424+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:29.748424+0000","last_peered":"2024-04-22T08:29:29.748424+0000","last_clean":"2024-04-22T08:29:29.748424+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:29.748424+0000","last_undegraded":"2024-04-22T08:29:29.748424+0000","last_fullsized":"2024-04-22T08:29:29.748424+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":16556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7045","reported_seq":7166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.755451+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:29.755451+0000","last_peered":"2024-04-22T08:29:29.755451+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:29.755451+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28839936,"num_objects":7041,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7090","reported_seq":7118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:29.662181+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:29.662181+0000","last_peered":"2024-04-22T08:29:29.662181+0000","last_clean":"2024-04-22T08:29:29.662181+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:29.662181+0000","last_undegraded":"2024-04-22T08:29:29.662181+0000","last_fullsized":"2024-04-22T08:29:29.662181+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7090,"log_dups_size":0,"ondisk_log_size":7090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29040640,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":28360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7213","reported_seq":7349,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.033292+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:34.033292+0000","last_peered":"2024-04-22T08:29:34.033292+0000","last_clean":"2024-04-22T08:29:34.033292+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:34.033292+0000","last_undegraded":"2024-04-22T08:29:34.033292+0000","last_fullsized":"2024-04-22T08:29:34.033292+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":17220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7317","reported_seq":7477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.059632+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:34.059632+0000","last_peered":"2024-04-22T08:29:34.059632+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:34.059632+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":16744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3433},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7115","reported_seq":12237,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:34.091333+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:34.091333+0000","last_peered":"2024-04-22T08:29:34.091333+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:34.091333+0000","last_undegraded":"2024-04-22T08:29:34.091333+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118464,"num_objects":7109,"num_object_clones":0,"num_object_copies":14218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2482,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7123","reported_seq":7265,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:33.951196+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:33.951196+0000","last_peered":"2024-04-22T08:29:33.951196+0000","last_clean":"2024-04-22T08:29:33.951196+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:33.951196+0000","last_undegraded":"2024-04-22T08:29:33.951196+0000","last_fullsized":"2024-04-22T08:29:33.951196+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7210","reported_seq":7330,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.721385+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:29.721385+0000","last_peered":"2024-04-22T08:29:29.721385+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:29.721385+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29267154,"num_objects":7147,"num_object_clones":0,"num_object_copies":14294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":28584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7233","reported_seq":7308,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.064623+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:34.064623+0000","last_peered":"2024-04-22T08:29:34.064623+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:34.064623+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594834,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7035","reported_seq":7179,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.708524+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:29.708524+0000","last_peered":"2024-04-22T08:29:29.708524+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:29.708524+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28794880,"num_objects":7030,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":13956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3280}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7060","reported_seq":7113,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:29.602810+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:29.602810+0000","last_peered":"2024-04-22T08:29:29.602810+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:29.602810+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28894418,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":28219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7195","reported_seq":7272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:33.998886+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:33.998886+0000","last_peered":"2024-04-22T08:29:33.998886+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:33.998886+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463145,"num_objects":7194,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":28773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3473},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7129","reported_seq":7301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.016285+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:34.016285+0000","last_peered":"2024-04-22T08:29:34.016285+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:34.016285+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3360,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3284},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7206","reported_seq":7280,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.060046+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:34.060046+0000","last_peered":"2024-04-22T08:29:34.060046+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:34.060046+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7206,"log_dups_size":0,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":28810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3499},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7267","reported_seq":7408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.085343+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:34.085343+0000","last_peered":"2024-04-22T08:29:34.085343+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:34.085343+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29718948,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7264,"num_write_kb":29027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3378},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":468852565,"num_objects":114477,"num_object_clones":0,"num_object_copies":228954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":114477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88079,"num_write_kb":352229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":947933184,"data_stored":947853167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114604,"ondisk_log_size":114604,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531412,"kb_used_data":347212,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93840428,"statfs":{"total":96636764160,"available":96092598272,"internally_reserved":0,"allocated":355545088,"data_stored":355344073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.4390000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920789,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355208,"kb_used_data":171008,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":94016632,"statfs":{"total":96636764160,"available":96273031168,"internally_reserved":0,"allocated":175112192,"data_stored":174841597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":188608204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,4,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.0209999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.32300000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672388,"kb_used_data":406204,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93699452,"statfs":{"total":96636764160,"available":95948238848,"internally_reserved":0,"allocated":415952896,"data_stored":415758343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.165},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":416518144,"data_stored":416514665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175579136,"data_stored":175506077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355835904,"data_stored":355832425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:34.864 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 142 20 115693 115673 3.18087 1.97656 0.0167851 0.0245539 2024-04-22T08:29:35.864 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 143 20 116231 116211 3.17332 2.10156 0.0123885 0.0246105 2024-04-22T08:29:36.864 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 144 20 116756 116736 3.16553 2.05078 0.0505454 0.0246743 2024-04-22T08:29:37.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:37.864 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 145 20 117274 117254 3.15765 2.02344 0.0117533 0.0247343 2024-04-22T08:29:38.333 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:38.423 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-22T08:29:38.189591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472780629,"num_objects":115436,"num_object_clones":0,"num_object_copies":230872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":115436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89038,"num_write_kb":356065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115563,"ondisk_log_size":115563,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1551940,"kb_used_data":937836,"kb_used_omap":12,"kb_used_meta":614067,"kb_avail":281563580,"statfs":{"total":289910292480,"available":288321105920,"internally_reserved":0,"allocated":960344064,"data_stored":959677901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":628804718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,11,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20258816,"num_objects":4946,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-305,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":19784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":1249280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.f","version":"35'7283","reported_seq":7344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.000525+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:34.000525+0000","last_peered":"2024-04-22T08:29:34.000525+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:34.000525+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29831168,"num_objects":7283,"num_object_clones":0,"num_object_copies":14566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":29132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3489},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7259","reported_seq":7365,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.749573+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:34.749573+0000","last_peered":"2024-04-22T08:29:34.749573+0000","last_clean":"2024-04-22T08:29:34.749573+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:34.749573+0000","last_undegraded":"2024-04-22T08:29:34.749573+0000","last_fullsized":"2024-04-22T08:29:34.749573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29708288,"num_objects":7253,"num_object_clones":0,"num_object_copies":14506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":17260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7204","reported_seq":7325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.765092+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:34.765092+0000","last_peered":"2024-04-22T08:29:34.765092+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:34.765092+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":7200,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":16124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7246","reported_seq":7274,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.732171+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:34.732171+0000","last_peered":"2024-04-22T08:29:34.732171+0000","last_clean":"2024-04-22T08:29:34.732171+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:34.732171+0000","last_undegraded":"2024-04-22T08:29:34.732171+0000","last_fullsized":"2024-04-22T08:29:34.732171+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":7246,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":28984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7213","reported_seq":7349,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.033292+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:34.033292+0000","last_peered":"2024-04-22T08:29:34.033292+0000","last_clean":"2024-04-22T08:29:34.033292+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:34.033292+0000","last_undegraded":"2024-04-22T08:29:34.033292+0000","last_fullsized":"2024-04-22T08:29:34.033292+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":17220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7317","reported_seq":7477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.059632+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:34.059632+0000","last_peered":"2024-04-22T08:29:34.059632+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:34.059632+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":16744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3433},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7115","reported_seq":12237,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:34.091333+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:34.091333+0000","last_peered":"2024-04-22T08:29:34.091333+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:34.091333+0000","last_undegraded":"2024-04-22T08:29:34.091333+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118464,"num_objects":7109,"num_object_clones":0,"num_object_copies":14218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2482,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7123","reported_seq":7265,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:33.951196+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:33.951196+0000","last_peered":"2024-04-22T08:29:33.951196+0000","last_clean":"2024-04-22T08:29:33.951196+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:33.951196+0000","last_undegraded":"2024-04-22T08:29:33.951196+0000","last_fullsized":"2024-04-22T08:29:33.951196+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7372","reported_seq":7492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.777130+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:34.777130+0000","last_peered":"2024-04-22T08:29:34.777130+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:34.777130+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7372,"log_dups_size":0,"ondisk_log_size":7372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29930706,"num_objects":7309,"num_object_clones":0,"num_object_copies":14618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7312,"num_write_kb":29232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7233","reported_seq":7308,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.064623+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:34.064623+0000","last_peered":"2024-04-22T08:29:34.064623+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:34.064623+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594834,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7197","reported_seq":7341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.744394+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:34.744394+0000","last_peered":"2024-04-22T08:29:34.744394+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:34.744394+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29458432,"num_objects":7192,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":14604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3442}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7204","reported_seq":7257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.726546+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:34.726546+0000","last_peered":"2024-04-22T08:29:34.726546+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:34.726546+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29484242,"num_objects":7200,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":28795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7195","reported_seq":7272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:33.998886+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:33.998886+0000","last_peered":"2024-04-22T08:29:33.998886+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:33.998886+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463145,"num_objects":7194,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":28773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3473},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7129","reported_seq":7301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.016285+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:34.016285+0000","last_peered":"2024-04-22T08:29:34.016285+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:34.016285+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3360,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3284},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7206","reported_seq":7280,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.060046+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:34.060046+0000","last_peered":"2024-04-22T08:29:34.060046+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:34.060046+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7206,"log_dups_size":0,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":28810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3499},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7267","reported_seq":7408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.085343+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:34.085343+0000","last_peered":"2024-04-22T08:29:34.085343+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:34.085343+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29718948,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7264,"num_write_kb":29027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3378},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":472780629,"num_objects":115436,"num_object_clones":0,"num_object_copies":230872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":115436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89038,"num_write_kb":356065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961728512,"data_stored":961648495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115563,"ondisk_log_size":115563,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117027,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524152,"kb_used_data":352240,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93847688,"statfs":{"total":96636764160,"available":96100032512,"internally_reserved":0,"allocated":360693760,"data_stored":360492745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":4.109}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.28000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920790,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355400,"kb_used_data":179392,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":94016440,"statfs":{"total":96636764160,"available":96272834560,"internally_reserved":0,"allocated":183697408,"data_stored":183426813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":180219596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.314}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.30499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672388,"kb_used_data":406204,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93699452,"statfs":{"total":96636764160,"available":95948238848,"internally_reserved":0,"allocated":415952896,"data_stored":415758343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.165},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":416518144,"data_stored":416514665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184180736,"data_stored":184107677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361029632,"data_stored":361026153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:38.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:38.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 146 20 117774 117754 3.14939 1.95312 0.0120702 0.0247978 2024-04-22T08:29:39.300 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:39.310 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:39.352 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-22T08:29:38.189591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472780629,"num_objects":115436,"num_object_clones":0,"num_object_copies":230872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":115436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89038,"num_write_kb":356065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115563,"ondisk_log_size":115563,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1551940,"kb_used_data":937836,"kb_used_omap":12,"kb_used_meta":614067,"kb_avail":281563580,"statfs":{"total":289910292480,"available":288321105920,"internally_reserved":0,"allocated":960344064,"data_stored":959677901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":628804718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,11,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20258816,"num_objects":4946,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-305,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":19784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":1249280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.f","version":"35'7283","reported_seq":7344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.000525+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:34.000525+0000","last_peered":"2024-04-22T08:29:34.000525+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:34.000525+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29831168,"num_objects":7283,"num_object_clones":0,"num_object_copies":14566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":29132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3489},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7259","reported_seq":7365,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.749573+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:34.749573+0000","last_peered":"2024-04-22T08:29:34.749573+0000","last_clean":"2024-04-22T08:29:34.749573+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:34.749573+0000","last_undegraded":"2024-04-22T08:29:34.749573+0000","last_fullsized":"2024-04-22T08:29:34.749573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29708288,"num_objects":7253,"num_object_clones":0,"num_object_copies":14506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":17260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7204","reported_seq":7325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.765092+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:34.765092+0000","last_peered":"2024-04-22T08:29:34.765092+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:34.765092+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":7200,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":16124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7246","reported_seq":7274,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.732171+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:34.732171+0000","last_peered":"2024-04-22T08:29:34.732171+0000","last_clean":"2024-04-22T08:29:34.732171+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:34.732171+0000","last_undegraded":"2024-04-22T08:29:34.732171+0000","last_fullsized":"2024-04-22T08:29:34.732171+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":7246,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":28984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7213","reported_seq":7349,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:34.033292+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:34.033292+0000","last_peered":"2024-04-22T08:29:34.033292+0000","last_clean":"2024-04-22T08:29:34.033292+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:34.033292+0000","last_undegraded":"2024-04-22T08:29:34.033292+0000","last_fullsized":"2024-04-22T08:29:34.033292+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29515776,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":17220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7317","reported_seq":7477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.059632+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:34.059632+0000","last_peered":"2024-04-22T08:29:34.059632+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:34.059632+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":16744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3433},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7115","reported_seq":12237,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:34.091333+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:34.091333+0000","last_peered":"2024-04-22T08:29:34.091333+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:34.091333+0000","last_undegraded":"2024-04-22T08:29:34.091333+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29118464,"num_objects":7109,"num_object_clones":0,"num_object_copies":14218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":13932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2482,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7123","reported_seq":7265,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:33.951196+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:33.951196+0000","last_peered":"2024-04-22T08:29:33.951196+0000","last_clean":"2024-04-22T08:29:33.951196+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:33.951196+0000","last_undegraded":"2024-04-22T08:29:33.951196+0000","last_fullsized":"2024-04-22T08:29:33.951196+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7372","reported_seq":7492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.777130+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:34.777130+0000","last_peered":"2024-04-22T08:29:34.777130+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:34.777130+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7372,"log_dups_size":0,"ondisk_log_size":7372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29930706,"num_objects":7309,"num_object_clones":0,"num_object_copies":14618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7312,"num_write_kb":29232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7233","reported_seq":7308,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.064623+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:34.064623+0000","last_peered":"2024-04-22T08:29:34.064623+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:34.064623+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594834,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7197","reported_seq":7341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.744394+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:34.744394+0000","last_peered":"2024-04-22T08:29:34.744394+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:34.744394+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29458432,"num_objects":7192,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":14604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3442}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7204","reported_seq":7257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.726546+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:34.726546+0000","last_peered":"2024-04-22T08:29:34.726546+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:34.726546+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29484242,"num_objects":7200,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":28795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7195","reported_seq":7272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:33.998886+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:33.998886+0000","last_peered":"2024-04-22T08:29:33.998886+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:33.998886+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463145,"num_objects":7194,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":28773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3473},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7129","reported_seq":7301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.016285+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:34.016285+0000","last_peered":"2024-04-22T08:29:34.016285+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:34.016285+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29167616,"num_objects":7121,"num_object_clones":0,"num_object_copies":14242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3360,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3284},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7206","reported_seq":7280,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.060046+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:34.060046+0000","last_peered":"2024-04-22T08:29:34.060046+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:34.060046+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7206,"log_dups_size":0,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":28810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3499},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7267","reported_seq":7408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:34.085343+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:34.085343+0000","last_peered":"2024-04-22T08:29:34.085343+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:34.085343+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29718948,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7264,"num_write_kb":29027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3378},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":472780629,"num_objects":115436,"num_object_clones":0,"num_object_copies":230872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1301,"num_objects_unfound":0,"num_objects_dirty":115436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89038,"num_write_kb":356065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":10175081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961728512,"data_stored":961648495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115563,"ondisk_log_size":115563,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117027,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524152,"kb_used_data":352240,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93847688,"statfs":{"total":96636764160,"available":96100032512,"internally_reserved":0,"allocated":360693760,"data_stored":360492745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":4.109}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.28000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920790,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355400,"kb_used_data":179392,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":94016440,"statfs":{"total":96636764160,"available":96272834560,"internally_reserved":0,"allocated":183697408,"data_stored":183426813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":180219596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.314}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.30499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313255,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672388,"kb_used_data":406204,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93699452,"statfs":{"total":96636764160,"available":95948238848,"internally_reserved":0,"allocated":415952896,"data_stored":415758343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":2.165},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":416518144,"data_stored":416514665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184180736,"data_stored":184107677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361029632,"data_stored":361026153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'7283", 'reported_seq': 7344, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.000525+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:34.000525+0000', 'last_peered': '2024-04-22T08:29:34.000525+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:34.000525+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7283, 'log_dups_size': 0, 'ondisk_log_size': 7283, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29831168, 'num_objects': 7283, 'num_object_clones': 0, 'num_object_copies': 14566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7283, 'num_write_kb': 29132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3489}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7204", 'reported_seq': 7325, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.765092+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:34.765092+0000', 'last_peered': '2024-04-22T08:29:34.765092+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:34.765092+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7204, 'log_dups_size': 0, 'ondisk_log_size': 7204, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29491200, 'num_objects': 7200, 'num_object_clones': 0, 'num_object_copies': 14400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4031, 'num_write_kb': 16124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 3445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:39.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'7317", 'reported_seq': 7477, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.059632+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:34.059632+0000', 'last_peered': '2024-04-22T08:29:34.059632+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:34.059632+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 7317, 'log_dups_size': 0, 'ondisk_log_size': 7317, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29949952, 'num_objects': 7312, 'num_object_clones': 0, 'num_object_copies': 14624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4186, 'num_write_kb': 16744, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3433}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'7115", 'reported_seq': 12237, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:34.091333+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:34.091333+0000', 'last_peered': '2024-04-22T08:29:34.091333+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:34.091333+0000', 'last_undegraded': '2024-04-22T08:29:34.091333+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 7115, 'log_dups_size': 0, 'ondisk_log_size': 7115, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29118464, 'num_objects': 7109, 'num_object_clones': 0, 'num_object_copies': 14218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1301, 'num_objects_unfound': 0, 'num_objects_dirty': 7109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3483, 'num_write_kb': 13932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2482, 'num_bytes_recovered': 10158080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5808}, {'shards': '0,2', 'objects': 1301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'7372", 'reported_seq': 7492, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.777130+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:34.777130+0000', 'last_peered': '2024-04-22T08:29:34.777130+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:34.777130+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 7372, 'log_dups_size': 0, 'ondisk_log_size': 7372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29930706, 'num_objects': 7309, 'num_object_clones': 0, 'num_object_copies': 14618, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7312, 'num_write_kb': 29232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 3421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'7233", 'reported_seq': 7308, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.064623+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:34.064623+0000', 'last_peered': '2024-04-22T08:29:34.064623+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:34.064623+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7233, 'log_dups_size': 0, 'ondisk_log_size': 7233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29594834, 'num_objects': 7227, 'num_object_clones': 0, 'num_object_copies': 14454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7231, 'num_write_kb': 28904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3456}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7197", 'reported_seq': 7341, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.744394+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:34.744394+0000', 'last_peered': '2024-04-22T08:29:34.744394+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:34.744394+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7197, 'log_dups_size': 0, 'ondisk_log_size': 7197, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29458432, 'num_objects': 7192, 'num_object_clones': 0, 'num_object_copies': 14384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3651, 'num_write_kb': 14604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 3442}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:39.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7204", 'reported_seq': 7257, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.726546+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:34.726546+0000', 'last_peered': '2024-04-22T08:29:34.726546+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:34.726546+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7204, 'log_dups_size': 0, 'ondisk_log_size': 7204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29484242, 'num_objects': 7200, 'num_object_clones': 0, 'num_object_copies': 14400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7202, 'num_write_kb': 28795, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 3397}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7195", 'reported_seq': 7272, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:33.998886+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:33.998886+0000', 'last_peered': '2024-04-22T08:29:33.998886+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:33.998886+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7195, 'log_dups_size': 0, 'ondisk_log_size': 7195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29463145, 'num_objects': 7194, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7194, 'num_write_kb': 28773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3473}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'7129", 'reported_seq': 7301, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.016285+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:34.016285+0000', 'last_peered': '2024-04-22T08:29:34.016285+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:34.016285+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 7129, 'log_dups_size': 0, 'ondisk_log_size': 7129, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29167616, 'num_objects': 7121, 'num_object_clones': 0, 'num_object_copies': 14242, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3360, 'num_write_kb': 13440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3284}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'7206", 'reported_seq': 7280, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.060046+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:34.060046+0000', 'last_peered': '2024-04-22T08:29:34.060046+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:34.060046+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7206, 'log_dups_size': 0, 'ondisk_log_size': 7206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29500626, 'num_objects': 7204, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7204, 'num_write_kb': 28810, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3499}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:39.356 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'7267", 'reported_seq': 7408, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:34.085343+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:34.085343+0000', 'last_peered': '2024-04-22T08:29:34.085343+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:34.085343+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 7267, 'log_dups_size': 0, 'ondisk_log_size': 7267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29718948, 'num_objects': 7259, 'num_object_clones': 0, 'num_object_copies': 14518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7264, 'num_write_kb': 29027, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3378}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:39.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:39.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 147 20 118247 118227 3.14053 1.84766 0.020627 0.0248674 2024-04-22T08:29:40.436 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:40.436 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/236076 objects degraded (17.708%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":215,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":118038,"data_bytes":483438421,"bytes_used":1610915840,"bytes_avail":288299376640,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":236076,"degraded_ratio":0.1770785679188058,"misplaced_objects":1148,"misplaced_total":236076,"misplaced_ratio":0.004862840780087768,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":156280,"recovering_keys_per_sec":0,"num_objects_recovered":458,"num_bytes_recovered":1875968,"num_keys_recovered":0,"write_bytes_sec":2575556,"read_op_per_sec":0,"write_op_per_sec":628},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:40.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:40.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:40.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 148 20 118737 118717 3.13225 1.91406 0.0536969 0.0249334 2024-04-22T08:29:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:41.325 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:41.365 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-04-22T08:29:40.190079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1573160,"kb_used_data":950864,"kb_used_omap":12,"kb_used_meta":622259,"kb_avail":281542360,"statfs":{"total":289910292480,"available":288299376640,"internally_reserved":0,"allocated":973684736,"data_stored":973018573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":637193326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,6,10,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30916608,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-458,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":30192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003853"},"pg_stats":[{"pgid":"2.f","version":"35'7465","reported_seq":7522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.149084+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:39.149084+0000","last_peered":"2024-04-22T08:29:39.149084+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:39.149084+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30576640,"num_objects":7465,"num_object_clones":0,"num_object_copies":14930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":29860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3671},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7450","reported_seq":7562,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.744237+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:39.744237+0000","last_peered":"2024-04-22T08:29:39.744237+0000","last_clean":"2024-04-22T08:29:39.744237+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:39.744237+0000","last_undegraded":"2024-04-22T08:29:39.744237+0000","last_fullsized":"2024-04-22T08:29:39.744237+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7450,"log_dups_size":0,"ondisk_log_size":7450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30490624,"num_objects":7444,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":18024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7333","reported_seq":7454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.755672+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:39.755672+0000","last_peered":"2024-04-22T08:29:39.755672+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:39.755672+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7333,"log_dups_size":0,"ondisk_log_size":7333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7396","reported_seq":7424,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.784326+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:39.784326+0000","last_peered":"2024-04-22T08:29:39.784326+0000","last_clean":"2024-04-22T08:29:39.784326+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:39.784326+0000","last_undegraded":"2024-04-22T08:29:39.784326+0000","last_fullsized":"2024-04-22T08:29:39.784326+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30294016,"num_objects":7396,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":29584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7368","reported_seq":7504,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.055006+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:39.055006+0000","last_peered":"2024-04-22T08:29:39.055006+0000","last_clean":"2024-04-22T08:29:39.055006+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:39.055006+0000","last_undegraded":"2024-04-22T08:29:39.055006+0000","last_fullsized":"2024-04-22T08:29:39.055006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30150656,"num_objects":7361,"num_object_clones":0,"num_object_copies":14722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":17840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7466","reported_seq":7626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.008232+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:39.008232+0000","last_peered":"2024-04-22T08:29:39.008232+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:39.008232+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7296","reported_seq":12726,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:39.148169+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:39.148169+0000","last_peered":"2024-04-22T08:29:39.148169+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:39.148169+0000","last_undegraded":"2024-04-22T08:29:39.148169+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29859840,"num_objects":7290,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":7290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":10784768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6142},{"shards":"0,2","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7293","reported_seq":7435,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.134468+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:39.134468+0000","last_peered":"2024-04-22T08:29:39.134468+0000","last_clean":"2024-04-22T08:29:39.134468+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:39.134468+0000","last_undegraded":"2024-04-22T08:29:39.134468+0000","last_fullsized":"2024-04-22T08:29:39.134468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29863936,"num_objects":7291,"num_object_clones":0,"num_object_copies":14582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":15764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7527","reported_seq":7647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.783348+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:39.783348+0000","last_peered":"2024-04-22T08:29:39.783348+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:39.783348+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30565586,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":29852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3576}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7397","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.112975+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:39.112975+0000","last_peered":"2024-04-22T08:29:39.112975+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:39.112975+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266578,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":29560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3620},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7351","reported_seq":7494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.832027+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:39.832027+0000","last_peered":"2024-04-22T08:29:39.832027+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:39.832027+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30089216,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":15220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7376","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.788865+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:39.788865+0000","last_peered":"2024-04-22T08:29:39.788865+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:39.788865+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30188754,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":29483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7354","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.140968+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:39.140968+0000","last_peered":"2024-04-22T08:29:39.140968+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:39.140968+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30114409,"num_objects":7353,"num_object_clones":0,"num_object_copies":14706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":29409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7293","reported_seq":7465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.156080+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:39.156080+0000","last_peered":"2024-04-22T08:29:39.156080+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:39.156080+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":14096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3448},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7389","reported_seq":7463,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.095098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:39.095098+0000","last_peered":"2024-04-22T08:29:39.095098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:39.095098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30250194,"num_objects":7387,"num_object_clones":0,"num_object_copies":14774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":29542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3682},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7411","reported_seq":7552,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.120527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:39.120527+0000","last_peered":"2024-04-22T08:29:39.120527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:39.120527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30308772,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3522},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":975179776,"data_stored":975099759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117028,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533468,"kb_used_data":357460,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93838372,"statfs":{"total":96636764160,"available":96090492928,"internally_reserved":0,"allocated":366039040,"data_stored":365838025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.5349999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.948},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.371}]}]},{"osd":1,"up_from":33,"seq":141733920790,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355400,"kb_used_data":179392,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":94016440,"statfs":{"total":96636764160,"available":96272834560,"internally_reserved":0,"allocated":183697408,"data_stored":183426813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":180219596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.314}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.30499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313256,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684292,"kb_used_data":414012,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93687548,"statfs":{"total":96636764160,"available":95936049152,"internally_reserved":0,"allocated":423948288,"data_stored":423753735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424722432,"data_stored":424718953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184180736,"data_stored":184107677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366276608,"data_stored":366273129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:41.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 149 20 119250 119230 3.12468 2.00391 0.0241308 0.0249956 2024-04-22T08:29:42.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 150 20 119757 119737 3.11705 1.98047 0.0350431 0.0250557 2024-04-22T08:29:43.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 151 20 120280 120260 3.10994 2.04297 0.0433642 0.025114 2024-04-22T08:29:44.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:44.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 152 20 120777 120757 3.10225 1.94141 0.0273648 0.0251727 2024-04-22T08:29:45.112 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:45.113 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-22T08:29:44.191278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1585316,"kb_used_data":958924,"kb_used_omap":12,"kb_used_meta":626355,"kb_avail":281530204,"statfs":{"total":289910292480,"available":288286928896,"internally_reserved":0,"allocated":981938176,"data_stored":981272013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":641387630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,4,7,7,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20918272,"num_objects":5107,"num_object_clones":0,"num_object_copies":10214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-306,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003269"},"pg_stats":[{"pgid":"2.f","version":"35'7465","reported_seq":7522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.149084+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:39.149084+0000","last_peered":"2024-04-22T08:29:39.149084+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:39.149084+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30576640,"num_objects":7465,"num_object_clones":0,"num_object_copies":14930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":29860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3671},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7450","reported_seq":7562,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.744237+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:39.744237+0000","last_peered":"2024-04-22T08:29:39.744237+0000","last_clean":"2024-04-22T08:29:39.744237+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:39.744237+0000","last_undegraded":"2024-04-22T08:29:39.744237+0000","last_fullsized":"2024-04-22T08:29:39.744237+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7450,"log_dups_size":0,"ondisk_log_size":7450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30490624,"num_objects":7444,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":18024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7333","reported_seq":7454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.755672+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:39.755672+0000","last_peered":"2024-04-22T08:29:39.755672+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:39.755672+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7333,"log_dups_size":0,"ondisk_log_size":7333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7396","reported_seq":7424,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.784326+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:39.784326+0000","last_peered":"2024-04-22T08:29:39.784326+0000","last_clean":"2024-04-22T08:29:39.784326+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:39.784326+0000","last_undegraded":"2024-04-22T08:29:39.784326+0000","last_fullsized":"2024-04-22T08:29:39.784326+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30294016,"num_objects":7396,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":29584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7368","reported_seq":7504,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.055006+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:39.055006+0000","last_peered":"2024-04-22T08:29:39.055006+0000","last_clean":"2024-04-22T08:29:39.055006+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:39.055006+0000","last_undegraded":"2024-04-22T08:29:39.055006+0000","last_fullsized":"2024-04-22T08:29:39.055006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30150656,"num_objects":7361,"num_object_clones":0,"num_object_copies":14722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":17840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7466","reported_seq":7626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.008232+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:39.008232+0000","last_peered":"2024-04-22T08:29:39.008232+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:39.008232+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7296","reported_seq":12726,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:39.148169+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:39.148169+0000","last_peered":"2024-04-22T08:29:39.148169+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:39.148169+0000","last_undegraded":"2024-04-22T08:29:39.148169+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29859840,"num_objects":7290,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":7290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":10784768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6142},{"shards":"0,2","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7293","reported_seq":7435,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.134468+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:39.134468+0000","last_peered":"2024-04-22T08:29:39.134468+0000","last_clean":"2024-04-22T08:29:39.134468+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:39.134468+0000","last_undegraded":"2024-04-22T08:29:39.134468+0000","last_fullsized":"2024-04-22T08:29:39.134468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29863936,"num_objects":7291,"num_object_clones":0,"num_object_copies":14582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":15764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7527","reported_seq":7647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.783348+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:39.783348+0000","last_peered":"2024-04-22T08:29:39.783348+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:39.783348+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30565586,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":29852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3576}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7397","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.112975+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:39.112975+0000","last_peered":"2024-04-22T08:29:39.112975+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:39.112975+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266578,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":29560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3620},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7351","reported_seq":7494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.832027+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:39.832027+0000","last_peered":"2024-04-22T08:29:39.832027+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:39.832027+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30089216,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":15220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7376","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.788865+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:39.788865+0000","last_peered":"2024-04-22T08:29:39.788865+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:39.788865+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30188754,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":29483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7354","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.140968+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:39.140968+0000","last_peered":"2024-04-22T08:29:39.140968+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:39.140968+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30114409,"num_objects":7353,"num_object_clones":0,"num_object_copies":14706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":29409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7293","reported_seq":7465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.156080+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:39.156080+0000","last_peered":"2024-04-22T08:29:39.156080+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:39.156080+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":14096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3448},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7389","reported_seq":7463,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.095098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:39.095098+0000","last_peered":"2024-04-22T08:29:39.095098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:39.095098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30250194,"num_objects":7387,"num_object_clones":0,"num_object_copies":14774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":29542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3682},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7411","reported_seq":7552,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.120527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:39.120527+0000","last_peered":"2024-04-22T08:29:39.120527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:39.120527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30308772,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3522},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983486464,"data_stored":983406447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117028,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533468,"kb_used_data":357460,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93838372,"statfs":{"total":96636764160,"available":96090492928,"internally_reserved":0,"allocated":366039040,"data_stored":365838025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.5349999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.948},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.371}]}]},{"osd":1,"up_from":33,"seq":141733920791,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367556,"kb_used_data":187452,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":94004284,"statfs":{"total":96636764160,"available":96260386816,"internally_reserved":0,"allocated":191950848,"data_stored":191680253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":184413900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":5.1210000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313256,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684292,"kb_used_data":414012,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93687548,"statfs":{"total":96636764160,"available":95936049152,"internally_reserved":0,"allocated":423948288,"data_stored":423753735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424722432,"data_stored":424718953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192487424,"data_stored":192414365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366276608,"data_stored":366273129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:45.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:45.865 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 153 20 121250 121230 3.09405 1.84766 0.0138033 0.0252394 2024-04-22T08:29:45.935 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:45.938 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:45.972 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-22T08:29:44.191278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1585316,"kb_used_data":958924,"kb_used_omap":12,"kb_used_meta":626355,"kb_avail":281530204,"statfs":{"total":289910292480,"available":288286928896,"internally_reserved":0,"allocated":981938176,"data_stored":981272013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":641387630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,4,7,7,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20918272,"num_objects":5107,"num_object_clones":0,"num_object_copies":10214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-306,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003269"},"pg_stats":[{"pgid":"2.f","version":"35'7465","reported_seq":7522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.149084+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:39.149084+0000","last_peered":"2024-04-22T08:29:39.149084+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:39.149084+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30576640,"num_objects":7465,"num_object_clones":0,"num_object_copies":14930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":29860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3671},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7450","reported_seq":7562,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.744237+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:39.744237+0000","last_peered":"2024-04-22T08:29:39.744237+0000","last_clean":"2024-04-22T08:29:39.744237+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:39.744237+0000","last_undegraded":"2024-04-22T08:29:39.744237+0000","last_fullsized":"2024-04-22T08:29:39.744237+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7450,"log_dups_size":0,"ondisk_log_size":7450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30490624,"num_objects":7444,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":18024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7333","reported_seq":7454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.755672+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:39.755672+0000","last_peered":"2024-04-22T08:29:39.755672+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:39.755672+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7333,"log_dups_size":0,"ondisk_log_size":7333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7396","reported_seq":7424,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.784326+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:39.784326+0000","last_peered":"2024-04-22T08:29:39.784326+0000","last_clean":"2024-04-22T08:29:39.784326+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:39.784326+0000","last_undegraded":"2024-04-22T08:29:39.784326+0000","last_fullsized":"2024-04-22T08:29:39.784326+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30294016,"num_objects":7396,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":29584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7368","reported_seq":7504,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.055006+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:39.055006+0000","last_peered":"2024-04-22T08:29:39.055006+0000","last_clean":"2024-04-22T08:29:39.055006+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:39.055006+0000","last_undegraded":"2024-04-22T08:29:39.055006+0000","last_fullsized":"2024-04-22T08:29:39.055006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30150656,"num_objects":7361,"num_object_clones":0,"num_object_copies":14722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":17840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7466","reported_seq":7626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.008232+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:39.008232+0000","last_peered":"2024-04-22T08:29:39.008232+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:39.008232+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7296","reported_seq":12726,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:39.148169+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:39.148169+0000","last_peered":"2024-04-22T08:29:39.148169+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:39.148169+0000","last_undegraded":"2024-04-22T08:29:39.148169+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29859840,"num_objects":7290,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":7290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":10784768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6142},{"shards":"0,2","objects":1148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7293","reported_seq":7435,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:39.134468+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:39.134468+0000","last_peered":"2024-04-22T08:29:39.134468+0000","last_clean":"2024-04-22T08:29:39.134468+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:39.134468+0000","last_undegraded":"2024-04-22T08:29:39.134468+0000","last_fullsized":"2024-04-22T08:29:39.134468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29863936,"num_objects":7291,"num_object_clones":0,"num_object_copies":14582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":15764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7527","reported_seq":7647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.783348+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:39.783348+0000","last_peered":"2024-04-22T08:29:39.783348+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:39.783348+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30565586,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":29852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3576}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7397","reported_seq":7471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.112975+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:39.112975+0000","last_peered":"2024-04-22T08:29:39.112975+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:39.112975+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266578,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":29560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3620},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7351","reported_seq":7494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.832027+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:39.832027+0000","last_peered":"2024-04-22T08:29:39.832027+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:39.832027+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30089216,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":15220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7376","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.788865+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:39.788865+0000","last_peered":"2024-04-22T08:29:39.788865+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:39.788865+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30188754,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":29483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7354","reported_seq":7429,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.140968+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:39.140968+0000","last_peered":"2024-04-22T08:29:39.140968+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:39.140968+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30114409,"num_objects":7353,"num_object_clones":0,"num_object_copies":14706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":29409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7293","reported_seq":7465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.156080+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:39.156080+0000","last_peered":"2024-04-22T08:29:39.156080+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:39.156080+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":14096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3448},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7389","reported_seq":7463,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.095098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:39.095098+0000","last_peered":"2024-04-22T08:29:39.095098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:39.095098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30250194,"num_objects":7387,"num_object_clones":0,"num_object_copies":14774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":29542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3682},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7411","reported_seq":7552,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:39.120527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:39.120527+0000","last_peered":"2024-04-22T08:29:39.120527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:39.120527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30308772,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3522},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483438421,"num_objects":118038,"num_object_clones":0,"num_object_copies":236076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":1148,"num_objects_unfound":0,"num_objects_dirty":118038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91640,"num_write_kb":366473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":10801769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983486464,"data_stored":983406447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118165,"ondisk_log_size":118165,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117028,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533468,"kb_used_data":357460,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93838372,"statfs":{"total":96636764160,"available":96090492928,"internally_reserved":0,"allocated":366039040,"data_stored":365838025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.5349999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.948},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.371}]}]},{"osd":1,"up_from":33,"seq":141733920791,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367556,"kb_used_data":187452,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":94004284,"statfs":{"total":96636764160,"available":96260386816,"internally_reserved":0,"allocated":191950848,"data_stored":191680253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":184413900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":5.1210000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313256,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684292,"kb_used_data":414012,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93687548,"statfs":{"total":96636764160,"available":95936049152,"internally_reserved":0,"allocated":423948288,"data_stored":423753735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":276754131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424722432,"data_stored":424718953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192487424,"data_stored":192414365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366276608,"data_stored":366273129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:45.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:45.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'7465", 'reported_seq': 7522, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.149084+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:39.149084+0000', 'last_peered': '2024-04-22T08:29:39.149084+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:39.149084+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7465, 'log_dups_size': 0, 'ondisk_log_size': 7465, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30576640, 'num_objects': 7465, 'num_object_clones': 0, 'num_object_copies': 14930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7465, 'num_write_kb': 29860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3671}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7333", 'reported_seq': 7454, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.755672+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:39.755672+0000', 'last_peered': '2024-04-22T08:29:39.755672+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:39.755672+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7333, 'log_dups_size': 0, 'ondisk_log_size': 7333, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30019584, 'num_objects': 7329, 'num_object_clones': 0, 'num_object_copies': 14658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4160, 'num_write_kb': 16640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 3574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'7466", 'reported_seq': 7626, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.008232+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:39.008232+0000', 'last_peered': '2024-04-22T08:29:39.008232+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:39.008232+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 7466, 'log_dups_size': 0, 'ondisk_log_size': 7466, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30560256, 'num_objects': 7461, 'num_object_clones': 0, 'num_object_copies': 14922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4335, 'num_write_kb': 17340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3582}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'7296", 'reported_seq': 12726, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:39.148169+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:39.148169+0000', 'last_peered': '2024-04-22T08:29:39.148169+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:39.148169+0000', 'last_undegraded': '2024-04-22T08:29:39.148169+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 7296, 'log_dups_size': 0, 'ondisk_log_size': 7296, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29859840, 'num_objects': 7290, 'num_object_clones': 0, 'num_object_copies': 14580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1148, 'num_objects_unfound': 0, 'num_objects_dirty': 7290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3664, 'num_write_kb': 14656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2635, 'num_bytes_recovered': 10784768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6142}, {'shards': '0,2', 'objects': 1148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:45.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'7527", 'reported_seq': 7647, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.783348+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:39.783348+0000', 'last_peered': '2024-04-22T08:29:39.783348+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:39.783348+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 7527, 'log_dups_size': 0, 'ondisk_log_size': 7527, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30565586, 'num_objects': 7464, 'num_object_clones': 0, 'num_object_copies': 14928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7467, 'num_write_kb': 29852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 3576}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'7397", 'reported_seq': 7471, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.112975+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:39.112975+0000', 'last_peered': '2024-04-22T08:29:39.112975+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:39.112975+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7397, 'log_dups_size': 0, 'ondisk_log_size': 7397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30266578, 'num_objects': 7391, 'num_object_clones': 0, 'num_object_copies': 14782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7395, 'num_write_kb': 29560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3620}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7351", 'reported_seq': 7494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.832027+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:39.832027+0000', 'last_peered': '2024-04-22T08:29:39.832027+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:39.832027+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7351, 'log_dups_size': 0, 'ondisk_log_size': 7351, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30089216, 'num_objects': 7346, 'num_object_clones': 0, 'num_object_copies': 14692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3805, 'num_write_kb': 15220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 3596}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7376", 'reported_seq': 7429, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.788865+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:39.788865+0000', 'last_peered': '2024-04-22T08:29:39.788865+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:39.788865+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7376, 'log_dups_size': 0, 'ondisk_log_size': 7376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30188754, 'num_objects': 7372, 'num_object_clones': 0, 'num_object_copies': 14744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7374, 'num_write_kb': 29483, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 3569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7354", 'reported_seq': 7429, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.140968+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:39.140968+0000', 'last_peered': '2024-04-22T08:29:39.140968+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:39.140968+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7354, 'log_dups_size': 0, 'ondisk_log_size': 7354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30114409, 'num_objects': 7353, 'num_object_clones': 0, 'num_object_copies': 14706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 29409, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3632}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:45.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'7293", 'reported_seq': 7465, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.156080+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:39.156080+0000', 'last_peered': '2024-04-22T08:29:39.156080+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:39.156080+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 7293, 'log_dups_size': 0, 'ondisk_log_size': 7293, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29839360, 'num_objects': 7285, 'num_object_clones': 0, 'num_object_copies': 14570, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3524, 'num_write_kb': 14096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3448}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:45.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'7389", 'reported_seq': 7463, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.095098+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:39.095098+0000', 'last_peered': '2024-04-22T08:29:39.095098+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:39.095098+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7389, 'log_dups_size': 0, 'ondisk_log_size': 7389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30250194, 'num_objects': 7387, 'num_object_clones': 0, 'num_object_copies': 14774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7387, 'num_write_kb': 29542, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3682}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:45.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'7411", 'reported_seq': 7552, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:39.120527+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:39.120527+0000', 'last_peered': '2024-04-22T08:29:39.120527+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:39.120527+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 7411, 'log_dups_size': 0, 'ondisk_log_size': 7411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30308772, 'num_objects': 7403, 'num_object_clones': 0, 'num_object_copies': 14806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7408, 'num_write_kb': 29603, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3522}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:45.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/236076 objects degraded (17.708%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":221,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":118038,"data_bytes":483438421,"bytes_used":1623363584,"bytes_avail":288286928896,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":236076,"degraded_ratio":0.1770785679188058,"misplaced_objects":1148,"misplaced_total":236076,"misplaced_ratio":0.004862840780087768,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":104419,"recovering_keys_per_sec":0,"num_objects_recovered":306,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":1742714,"read_op_per_sec":0,"write_op_per_sec":425},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:46.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:46.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:46.867 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 154 19 121628 121609 3.08355 1.48047 0.0527976 0.0253298 2024-04-22T08:29:47.174 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:47.174 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:47.219 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-04-22T08:29:46.191711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493825877,"num_objects":120574,"num_object_clones":0,"num_object_copies":241148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":968,"num_objects_unfound":0,"num_objects_dirty":120574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94176,"num_write_kb":376617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2835,"num_bytes_recovered":11539049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120701,"ondisk_log_size":120701,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1593768,"kb_used_data":971472,"kb_used_omap":12,"kb_used_meta":622259,"kb_avail":281521752,"statfs":{"total":289910292480,"available":288278274048,"internally_reserved":0,"allocated":994787328,"data_stored":994121165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":637193326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,13,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24973312,"num_objects":6097,"num_object_clones":0,"num_object_copies":12194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-333,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":24388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":333,"num_bytes_recovered":1363968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003177"},"pg_stats":[{"pgid":"2.f","version":"35'7609","reported_seq":7670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.213531+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:44.213531+0000","last_peered":"2024-04-22T08:29:44.213531+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:44.213531+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31166464,"num_objects":7609,"num_object_clones":0,"num_object_copies":15218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":30436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3815},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7600","reported_seq":7712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:44.773151+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:44.773151+0000","last_peered":"2024-04-22T08:29:44.773151+0000","last_clean":"2024-04-22T08:29:44.773151+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:44.773151+0000","last_undegraded":"2024-04-22T08:29:44.773151+0000","last_fullsized":"2024-04-22T08:29:44.773151+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31105024,"num_objects":7594,"num_object_clones":0,"num_object_copies":15188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7483","reported_seq":7604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.722966+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:44.722966+0000","last_peered":"2024-04-22T08:29:44.722966+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:44.722966+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30633984,"num_objects":7479,"num_object_clones":0,"num_object_copies":14958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":17240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3724}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7590","reported_seq":7618,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:44.851682+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:44.851682+0000","last_peered":"2024-04-22T08:29:44.851682+0000","last_clean":"2024-04-22T08:29:44.851682+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:44.851682+0000","last_undegraded":"2024-04-22T08:29:44.851682+0000","last_fullsized":"2024-04-22T08:29:44.851682+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7590,"log_dups_size":0,"ondisk_log_size":7590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088640,"num_objects":7590,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7590,"num_write_kb":30360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7526","reported_seq":7662,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:44.111003+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:44.111003+0000","last_peered":"2024-04-22T08:29:44.111003+0000","last_clean":"2024-04-22T08:29:44.111003+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:44.111003+0000","last_undegraded":"2024-04-22T08:29:44.111003+0000","last_fullsized":"2024-04-22T08:29:44.111003+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7526,"log_dups_size":0,"ondisk_log_size":7526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30797824,"num_objects":7519,"num_object_clones":0,"num_object_copies":15038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":18472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7621","reported_seq":7781,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.163076+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:44.163076+0000","last_peered":"2024-04-22T08:29:44.163076+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:44.163076+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7621,"log_dups_size":0,"ondisk_log_size":7621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":7616,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":17960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3737},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7441","reported_seq":13231,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:44.187276+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:44.187276+0000","last_peered":"2024-04-22T08:29:44.187276+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:44.187276+0000","last_undegraded":"2024-04-22T08:29:44.187276+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30453760,"num_objects":7435,"num_object_clones":0,"num_object_copies":14870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":968,"num_objects_unfound":0,"num_objects_dirty":7435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":15236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":11522048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6467},{"shards":"0,2","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7451","reported_seq":7593,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:44.133851+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:44.133851+0000","last_peered":"2024-04-22T08:29:44.133851+0000","last_clean":"2024-04-22T08:29:44.133851+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:44.133851+0000","last_undegraded":"2024-04-22T08:29:44.133851+0000","last_fullsized":"2024-04-22T08:29:44.133851+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7451,"log_dups_size":0,"ondisk_log_size":7451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30511104,"num_objects":7449,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":16396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7690","reported_seq":7810,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.790081+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:44.790081+0000","last_peered":"2024-04-22T08:29:44.790081+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:44.790081+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31233234,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":30504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7565","reported_seq":7640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.140355+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:44.140355+0000","last_peered":"2024-04-22T08:29:44.140355+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:44.140355+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7565,"log_dups_size":0,"ondisk_log_size":7565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954706,"num_objects":7559,"num_object_clones":0,"num_object_copies":15118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":30232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3788},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7493","reported_seq":7637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.784002+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:44.784002+0000","last_peered":"2024-04-22T08:29:44.784002+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:44.784002+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7493,"log_dups_size":0,"ondisk_log_size":7493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3947,"num_write_kb":15788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3738}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7536","reported_seq":7583,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.839944+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:44.839944+0000","last_peered":"2024-04-22T08:29:44.839944+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:44.839944+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30844114,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7534,"num_write_kb":30123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3729}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7515","reported_seq":7592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.212631+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:44.212631+0000","last_peered":"2024-04-22T08:29:44.212631+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:44.212631+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30773865,"num_objects":7514,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":30053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3793},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7452","reported_seq":7624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.220653+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:44.220653+0000","last_peered":"2024-04-22T08:29:44.220653+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:44.220653+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7452,"log_dups_size":0,"ondisk_log_size":7452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30490624,"num_objects":7444,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3683,"num_write_kb":14732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3607},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7551","reported_seq":7625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.136641+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:44.136641+0000","last_peered":"2024-04-22T08:29:44.136641+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:44.136641+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30913746,"num_objects":7549,"num_object_clones":0,"num_object_copies":15098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":30190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3844},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7578","reported_seq":7719,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:44.097723+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:44.097723+0000","last_peered":"2024-04-22T08:29:44.097723+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:44.097723+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30992804,"num_objects":7570,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":30271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3689},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":493825877,"num_objects":120574,"num_object_clones":0,"num_object_copies":241148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":968,"num_objects_unfound":0,"num_objects_dirty":120574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94176,"num_write_kb":376617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2835,"num_bytes_recovered":11539049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":996556800,"data_stored":996476783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120701,"ondisk_log_size":120701,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542588,"kb_used_data":362484,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93829252,"statfs":{"total":96636764160,"available":96081154048,"internally_reserved":0,"allocated":371183616,"data_stored":370982601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184413903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.307}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.26000000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920791,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367556,"kb_used_data":187452,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":94004284,"statfs":{"total":96636764160,"available":96260386816,"internally_reserved":0,"allocated":191950848,"data_stored":191680253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":184413900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":5.1210000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313257,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683624,"kb_used_data":421536,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93688216,"statfs":{"total":96636764160,"available":95936733184,"internally_reserved":0,"allocated":431652864,"data_stored":431458311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":268365523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":432607232,"data_stored":432603753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192487424,"data_stored":192414365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371462144,"data_stored":371458665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:47.867 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 155 20 122119 122099 3.076 1.91406 0.0105725 0.0253883 2024-04-22T08:29:48.867 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 156 20 122638 122618 3.06928 2.02734 0.0269393 0.0254457 2024-04-22T08:29:49.867 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 157 20 123143 123123 3.0623 1.97266 0.0296835 0.0255046 2024-04-22T08:29:50.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:50.867 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 158 20 123618 123598 3.05466 1.85547 0.0584552 0.0255681 2024-04-22T08:29:51.065 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:51.107 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-04-22T08:29:50.193738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613560,"kb_used_data":991264,"kb_used_omap":12,"kb_used_meta":622259,"kb_avail":281501960,"statfs":{"total":289910292480,"available":288258007040,"internally_reserved":0,"allocated":1015054336,"data_stored":1014388173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":637193326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,7,13,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30765056,"num_objects":7511,"num_object_clones":0,"num_object_copies":15022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-519,"num_objects_unfound":0,"num_objects_dirty":7511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":30044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":2125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004147"},"pg_stats":[{"pgid":"2.f","version":"35'7754","reported_seq":7815,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.237848+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:49.237848+0000","last_peered":"2024-04-22T08:29:49.237848+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:49.237848+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31760384,"num_objects":7754,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":31016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3960},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7758","reported_seq":7870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.838268+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:49.838268+0000","last_peered":"2024-04-22T08:29:49.838268+0000","last_clean":"2024-04-22T08:29:49.838268+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:49.838268+0000","last_undegraded":"2024-04-22T08:29:49.838268+0000","last_fullsized":"2024-04-22T08:29:49.838268+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31752192,"num_objects":7752,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":19256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7625","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.864102+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:49.864102+0000","last_peered":"2024-04-22T08:29:49.864102+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:49.864102+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7625,"log_dups_size":0,"ondisk_log_size":7625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31215616,"num_objects":7621,"num_object_clones":0,"num_object_copies":15242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7733","reported_seq":7761,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.759459+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:49.759459+0000","last_peered":"2024-04-22T08:29:49.759459+0000","last_clean":"2024-04-22T08:29:49.759459+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:49.759459+0000","last_undegraded":"2024-04-22T08:29:49.759459+0000","last_fullsized":"2024-04-22T08:29:49.759459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31674368,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":30932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7689","reported_seq":7825,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215850+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:49.215850+0000","last_peered":"2024-04-22T08:29:49.215850+0000","last_clean":"2024-04-22T08:29:49.215850+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:49.215850+0000","last_undegraded":"2024-04-22T08:29:49.215850+0000","last_fullsized":"2024-04-22T08:29:49.215850+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":19124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7788","reported_seq":7946,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.215576+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:49.215576+0000","last_peered":"2024-04-22T08:29:49.215576+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:49.215576+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31879168,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":18628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3904},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7589","reported_seq":13751,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:49.259361+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:49.259361+0000","last_peered":"2024-04-22T08:29:49.259361+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:49.259361+0000","last_undegraded":"2024-04-22T08:29:49.259361+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31059968,"num_objects":7583,"num_object_clones":0,"num_object_copies":15166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":7583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":15828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3001,"num_bytes_recovered":12283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6801},{"shards":"0,2","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7583","reported_seq":7725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215847+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:49.215847+0000","last_peered":"2024-04-22T08:29:49.215847+0000","last_clean":"2024-04-22T08:29:49.215847+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:49.215847+0000","last_undegraded":"2024-04-22T08:29:49.215847+0000","last_fullsized":"2024-04-22T08:29:49.215847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7827","reported_seq":7948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.872594+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:49.872594+0000","last_peered":"2024-04-22T08:29:49.872594+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:49.872594+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31794386,"num_objects":7764,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7767,"num_write_kb":31052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7719","reported_seq":7794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.174955+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:49.174955+0000","last_peered":"2024-04-22T08:29:49.174955+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:49.174955+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31585490,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3942},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7647","reported_seq":7791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.659405+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:49.659405+0000","last_peered":"2024-04-22T08:29:49.659405+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:49.659405+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31301632,"num_objects":7642,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4101,"num_write_kb":16404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7683","reported_seq":7736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.861601+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:49.861601+0000","last_peered":"2024-04-22T08:29:49.861601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:49.861601+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7683,"log_dups_size":0,"ondisk_log_size":7683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31446226,"num_objects":7679,"num_object_clones":0,"num_object_copies":15358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7681,"num_write_kb":30711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7668","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.288639+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:49.288639+0000","last_peered":"2024-04-22T08:29:49.288639+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:49.288639+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31400553,"num_objects":7667,"num_object_clones":0,"num_object_copies":15334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":30665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3946},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7589","reported_seq":7760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.236961+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:49.236961+0000","last_peered":"2024-04-22T08:29:49.236961+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:49.236961+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":15280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3744},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7692","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.239293+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:49.239293+0000","last_peered":"2024-04-22T08:29:49.239293+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:49.239293+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31491282,"num_objects":7690,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":30754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7730","reported_seq":7870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.268042+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:49.268042+0000","last_peered":"2024-04-22T08:29:49.268042+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:49.268042+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31615396,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":30879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3841},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1016721408,"data_stored":1016641391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551476,"kb_used_data":367276,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93820364,"statfs":{"total":96636764160,"available":96072052736,"internally_reserved":0,"allocated":376090624,"data_stored":375889609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":7.024},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":7.0579999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.24}]}]},{"osd":1,"up_from":33,"seq":141733920792,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367172,"kb_used_data":195260,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94004668,"statfs":{"total":96636764160,"available":96260780032,"internally_reserved":0,"allocated":199946240,"data_stored":199675645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":176025292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.20899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.25}]}]},{"osd":0,"up_from":21,"seq":90194313258,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694912,"kb_used_data":428728,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93676928,"statfs":{"total":96636764160,"available":95925174272,"internally_reserved":0,"allocated":439017472,"data_stored":438822919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":439975936,"data_stored":439972457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200519680,"data_stored":200446621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376225792,"data_stored":376222313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:51.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:51.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 159 20 124090 124070 3.04701 1.84375 0.0135844 0.0256306 2024-04-22T08:29:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:51.996 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:52.064 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-04-22T08:29:50.193738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613560,"kb_used_data":991264,"kb_used_omap":12,"kb_used_meta":622259,"kb_avail":281501960,"statfs":{"total":289910292480,"available":288258007040,"internally_reserved":0,"allocated":1015054336,"data_stored":1014388173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":637193326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,7,13,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30765056,"num_objects":7511,"num_object_clones":0,"num_object_copies":15022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-519,"num_objects_unfound":0,"num_objects_dirty":7511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":30044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":2125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004147"},"pg_stats":[{"pgid":"2.f","version":"35'7754","reported_seq":7815,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.237848+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:49.237848+0000","last_peered":"2024-04-22T08:29:49.237848+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:49.237848+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31760384,"num_objects":7754,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":31016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3960},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7758","reported_seq":7870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.838268+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:49.838268+0000","last_peered":"2024-04-22T08:29:49.838268+0000","last_clean":"2024-04-22T08:29:49.838268+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:49.838268+0000","last_undegraded":"2024-04-22T08:29:49.838268+0000","last_fullsized":"2024-04-22T08:29:49.838268+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31752192,"num_objects":7752,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":19256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7625","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.864102+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:49.864102+0000","last_peered":"2024-04-22T08:29:49.864102+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:49.864102+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7625,"log_dups_size":0,"ondisk_log_size":7625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31215616,"num_objects":7621,"num_object_clones":0,"num_object_copies":15242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7733","reported_seq":7761,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.759459+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:49.759459+0000","last_peered":"2024-04-22T08:29:49.759459+0000","last_clean":"2024-04-22T08:29:49.759459+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:49.759459+0000","last_undegraded":"2024-04-22T08:29:49.759459+0000","last_fullsized":"2024-04-22T08:29:49.759459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31674368,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":30932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7689","reported_seq":7825,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215850+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:49.215850+0000","last_peered":"2024-04-22T08:29:49.215850+0000","last_clean":"2024-04-22T08:29:49.215850+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:49.215850+0000","last_undegraded":"2024-04-22T08:29:49.215850+0000","last_fullsized":"2024-04-22T08:29:49.215850+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":19124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7788","reported_seq":7946,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.215576+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:49.215576+0000","last_peered":"2024-04-22T08:29:49.215576+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:49.215576+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31879168,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":18628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3904},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7589","reported_seq":13751,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:49.259361+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:49.259361+0000","last_peered":"2024-04-22T08:29:49.259361+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:49.259361+0000","last_undegraded":"2024-04-22T08:29:49.259361+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31059968,"num_objects":7583,"num_object_clones":0,"num_object_copies":15166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":7583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":15828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3001,"num_bytes_recovered":12283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6801},{"shards":"0,2","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7583","reported_seq":7725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215847+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:49.215847+0000","last_peered":"2024-04-22T08:29:49.215847+0000","last_clean":"2024-04-22T08:29:49.215847+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:49.215847+0000","last_undegraded":"2024-04-22T08:29:49.215847+0000","last_fullsized":"2024-04-22T08:29:49.215847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7827","reported_seq":7948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.872594+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:49.872594+0000","last_peered":"2024-04-22T08:29:49.872594+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:49.872594+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31794386,"num_objects":7764,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7767,"num_write_kb":31052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7719","reported_seq":7794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.174955+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:49.174955+0000","last_peered":"2024-04-22T08:29:49.174955+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:49.174955+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31585490,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3942},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7647","reported_seq":7791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.659405+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:49.659405+0000","last_peered":"2024-04-22T08:29:49.659405+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:49.659405+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31301632,"num_objects":7642,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4101,"num_write_kb":16404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7683","reported_seq":7736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.861601+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:49.861601+0000","last_peered":"2024-04-22T08:29:49.861601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:49.861601+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7683,"log_dups_size":0,"ondisk_log_size":7683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31446226,"num_objects":7679,"num_object_clones":0,"num_object_copies":15358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7681,"num_write_kb":30711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7668","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.288639+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:49.288639+0000","last_peered":"2024-04-22T08:29:49.288639+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:49.288639+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31400553,"num_objects":7667,"num_object_clones":0,"num_object_copies":15334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":30665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3946},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7589","reported_seq":7760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.236961+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:49.236961+0000","last_peered":"2024-04-22T08:29:49.236961+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:49.236961+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":15280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3744},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7692","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.239293+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:49.239293+0000","last_peered":"2024-04-22T08:29:49.239293+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:49.239293+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31491282,"num_objects":7690,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":30754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7730","reported_seq":7870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.268042+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:49.268042+0000","last_peered":"2024-04-22T08:29:49.268042+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:49.268042+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31615396,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":30879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3841},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1016721408,"data_stored":1016641391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551476,"kb_used_data":367276,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93820364,"statfs":{"total":96636764160,"available":96072052736,"internally_reserved":0,"allocated":376090624,"data_stored":375889609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":7.024},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":7.0579999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.24}]}]},{"osd":1,"up_from":33,"seq":141733920792,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367172,"kb_used_data":195260,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94004668,"statfs":{"total":96636764160,"available":96260780032,"internally_reserved":0,"allocated":199946240,"data_stored":199675645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":176025292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.20899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.25}]}]},{"osd":0,"up_from":21,"seq":90194313258,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694912,"kb_used_data":428728,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93676928,"statfs":{"total":96636764160,"available":95925174272,"internally_reserved":0,"allocated":439017472,"data_stored":438822919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":439975936,"data_stored":439972457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200519680,"data_stored":200446621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376225792,"data_stored":376222313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:52.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:52.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'7754", 'reported_seq': 7815, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.237848+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:49.237848+0000', 'last_peered': '2024-04-22T08:29:49.237848+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:49.237848+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7754, 'log_dups_size': 0, 'ondisk_log_size': 7754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31760384, 'num_objects': 7754, 'num_object_clones': 0, 'num_object_copies': 15508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7754, 'num_write_kb': 31016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3960}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:52.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7625", 'reported_seq': 7745, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.864102+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:49.864102+0000', 'last_peered': '2024-04-22T08:29:49.864102+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:49.864102+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7625, 'log_dups_size': 0, 'ondisk_log_size': 7625, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31215616, 'num_objects': 7621, 'num_object_clones': 0, 'num_object_copies': 15242, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4452, 'num_write_kb': 17808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 3866}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'7788", 'reported_seq': 7946, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.215576+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:49.215576+0000', 'last_peered': '2024-04-22T08:29:49.215576+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:49.215576+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31879168, 'num_objects': 7783, 'num_object_clones': 0, 'num_object_copies': 15566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4657, 'num_write_kb': 18628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3904}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'7589", 'reported_seq': 13751, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:49.259361+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:49.259361+0000', 'last_peered': '2024-04-22T08:29:49.259361+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:49.259361+0000', 'last_undegraded': '2024-04-22T08:29:49.259361+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 7589, 'log_dups_size': 0, 'ondisk_log_size': 7589, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31059968, 'num_objects': 7583, 'num_object_clones': 0, 'num_object_copies': 15166, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 782, 'num_objects_unfound': 0, 'num_objects_dirty': 7583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3957, 'num_write_kb': 15828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3001, 'num_bytes_recovered': 12283904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6801}, {'shards': '0,2', 'objects': 782}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'7827", 'reported_seq': 7948, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.872594+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:49.872594+0000', 'last_peered': '2024-04-22T08:29:49.872594+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:49.872594+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 7827, 'log_dups_size': 0, 'ondisk_log_size': 7827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31794386, 'num_objects': 7764, 'num_object_clones': 0, 'num_object_copies': 15528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7767, 'num_write_kb': 31052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 3876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'7719", 'reported_seq': 7794, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.174955+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:49.174955+0000', 'last_peered': '2024-04-22T08:29:49.174955+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:49.174955+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7719, 'log_dups_size': 0, 'ondisk_log_size': 7719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31585490, 'num_objects': 7713, 'num_object_clones': 0, 'num_object_copies': 15426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7717, 'num_write_kb': 30848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3942}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7647", 'reported_seq': 7791, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.659405+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:49.659405+0000', 'last_peered': '2024-04-22T08:29:49.659405+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:49.659405+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7647, 'log_dups_size': 0, 'ondisk_log_size': 7647, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31301632, 'num_objects': 7642, 'num_object_clones': 0, 'num_object_copies': 15284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4101, 'num_write_kb': 16404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 3892}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:52.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7683", 'reported_seq': 7736, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.861601+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:49.861601+0000', 'last_peered': '2024-04-22T08:29:49.861601+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:49.861601+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7683, 'log_dups_size': 0, 'ondisk_log_size': 7683, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31446226, 'num_objects': 7679, 'num_object_clones': 0, 'num_object_copies': 15358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7681, 'num_write_kb': 30711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 3876}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7668", 'reported_seq': 7745, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.288639+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:49.288639+0000', 'last_peered': '2024-04-22T08:29:49.288639+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:49.288639+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7668, 'log_dups_size': 0, 'ondisk_log_size': 7668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31400553, 'num_objects': 7667, 'num_object_clones': 0, 'num_object_copies': 15334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7667, 'num_write_kb': 30665, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3946}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'7589", 'reported_seq': 7760, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.236961+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:49.236961+0000', 'last_peered': '2024-04-22T08:29:49.236961+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:49.236961+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 7589, 'log_dups_size': 0, 'ondisk_log_size': 7589, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31051776, 'num_objects': 7581, 'num_object_clones': 0, 'num_object_copies': 15162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3820, 'num_write_kb': 15280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3744}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'7692", 'reported_seq': 7766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.239293+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:49.239293+0000', 'last_peered': '2024-04-22T08:29:49.239293+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:49.239293+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7692, 'log_dups_size': 0, 'ondisk_log_size': 7692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31491282, 'num_objects': 7690, 'num_object_clones': 0, 'num_object_copies': 15380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7690, 'num_write_kb': 30754, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3985}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:52.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'7730", 'reported_seq': 7870, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:49.268042+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:49.268042+0000', 'last_peered': '2024-04-22T08:29:49.268042+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:49.268042+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 7730, 'log_dups_size': 0, 'ondisk_log_size': 7730, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31615396, 'num_objects': 7722, 'num_object_clones': 0, 'num_object_copies': 15444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7727, 'num_write_kb': 30879, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3841}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:52.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:52.869 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:29:52.871535+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0256959 2024-04-22T08:29:52.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:29:52.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 160 20 124557 124537 3.03936 1.82422 0.0263228 0.0256959 2024-04-22T08:29:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:53.148 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/245894 objects degraded (17.001%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":228,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":122947,"data_bytes":503545685,"bytes_used":1652285440,"bytes_avail":288258007040,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":245894,"degraded_ratio":0.1700082149218769,"misplaced_objects":782,"misplaced_total":245894,"misplaced_ratio":0.003180232132544918,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":124871,"recovering_keys_per_sec":0,"num_objects_recovered":366,"num_bytes_recovered":1499136,"num_keys_recovered":0,"write_bytes_sec":1674853,"read_op_per_sec":0,"write_op_per_sec":408},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:29:53.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:29:53.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:53.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 161 20 125024 125004 3.03182 1.82422 0.0167395 0.0257612 2024-04-22T08:29:54.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:54.035 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:54.080 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-04-22T08:29:52.195463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613560,"kb_used_data":991264,"kb_used_omap":12,"kb_used_meta":622259,"kb_avail":281501960,"statfs":{"total":289910292480,"available":288258007040,"internally_reserved":0,"allocated":1015054336,"data_stored":1014388173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":637193326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,7,13,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20107264,"num_objects":4909,"num_object_clones":0,"num_object_copies":9818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-366,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":19636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":1499136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005383"},"pg_stats":[{"pgid":"2.f","version":"35'7754","reported_seq":7815,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.237848+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:49.237848+0000","last_peered":"2024-04-22T08:29:49.237848+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:49.237848+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31760384,"num_objects":7754,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":31016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3960},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7758","reported_seq":7870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.838268+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:49.838268+0000","last_peered":"2024-04-22T08:29:49.838268+0000","last_clean":"2024-04-22T08:29:49.838268+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:49.838268+0000","last_undegraded":"2024-04-22T08:29:49.838268+0000","last_fullsized":"2024-04-22T08:29:49.838268+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31752192,"num_objects":7752,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":19256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7625","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.864102+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:49.864102+0000","last_peered":"2024-04-22T08:29:49.864102+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:49.864102+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7625,"log_dups_size":0,"ondisk_log_size":7625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31215616,"num_objects":7621,"num_object_clones":0,"num_object_copies":15242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":3866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7733","reported_seq":7761,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.759459+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:49.759459+0000","last_peered":"2024-04-22T08:29:49.759459+0000","last_clean":"2024-04-22T08:29:49.759459+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:49.759459+0000","last_undegraded":"2024-04-22T08:29:49.759459+0000","last_fullsized":"2024-04-22T08:29:49.759459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31674368,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":30932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7689","reported_seq":7825,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215850+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:49.215850+0000","last_peered":"2024-04-22T08:29:49.215850+0000","last_clean":"2024-04-22T08:29:49.215850+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:49.215850+0000","last_undegraded":"2024-04-22T08:29:49.215850+0000","last_fullsized":"2024-04-22T08:29:49.215850+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":19124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7788","reported_seq":7946,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.215576+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:49.215576+0000","last_peered":"2024-04-22T08:29:49.215576+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:49.215576+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31879168,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":18628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3904},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7589","reported_seq":13751,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:49.259361+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:49.259361+0000","last_peered":"2024-04-22T08:29:49.259361+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:49.259361+0000","last_undegraded":"2024-04-22T08:29:49.259361+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31059968,"num_objects":7583,"num_object_clones":0,"num_object_copies":15166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":7583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3957,"num_write_kb":15828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3001,"num_bytes_recovered":12283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6801},{"shards":"0,2","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7583","reported_seq":7725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:49.215847+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:49.215847+0000","last_peered":"2024-04-22T08:29:49.215847+0000","last_clean":"2024-04-22T08:29:49.215847+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:49.215847+0000","last_undegraded":"2024-04-22T08:29:49.215847+0000","last_fullsized":"2024-04-22T08:29:49.215847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.106880+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.106880+0000","last_peered":"2024-04-22T08:27:56.106880+0000","last_clean":"2024-04-22T08:27:56.106880+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.106880+0000","last_undegraded":"2024-04-22T08:27:56.106880+0000","last_fullsized":"2024-04-22T08:27:56.106880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7827","reported_seq":7948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.872594+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:49.872594+0000","last_peered":"2024-04-22T08:29:49.872594+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:49.872594+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31794386,"num_objects":7764,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7767,"num_write_kb":31052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7719","reported_seq":7794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.174955+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:49.174955+0000","last_peered":"2024-04-22T08:29:49.174955+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:49.174955+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31585490,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3942},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7647","reported_seq":7791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.659405+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:49.659405+0000","last_peered":"2024-04-22T08:29:49.659405+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:49.659405+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31301632,"num_objects":7642,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4101,"num_write_kb":16404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7683","reported_seq":7736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.861601+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:49.861601+0000","last_peered":"2024-04-22T08:29:49.861601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:49.861601+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7683,"log_dups_size":0,"ondisk_log_size":7683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31446226,"num_objects":7679,"num_object_clones":0,"num_object_copies":15358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7681,"num_write_kb":30711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7668","reported_seq":7745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.288639+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:49.288639+0000","last_peered":"2024-04-22T08:29:49.288639+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:49.288639+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31400553,"num_objects":7667,"num_object_clones":0,"num_object_copies":15334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":30665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3946},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7589","reported_seq":7760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.236961+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:49.236961+0000","last_peered":"2024-04-22T08:29:49.236961+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:49.236961+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31051776,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":15280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3744},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7692","reported_seq":7766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.239293+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:49.239293+0000","last_peered":"2024-04-22T08:29:49.239293+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:49.239293+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31491282,"num_objects":7690,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":30754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7730","reported_seq":7870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:49.268042+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:49.268042+0000","last_peered":"2024-04-22T08:29:49.268042+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:49.268042+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31615396,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":30879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3841},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":503545685,"num_objects":122947,"num_object_clones":0,"num_object_copies":245894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":782,"num_objects_unfound":0,"num_objects_dirty":122947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96549,"num_write_kb":386109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3021,"num_bytes_recovered":12300905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1016721408,"data_stored":1016641391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123074,"ondisk_log_size":123074,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551476,"kb_used_data":367276,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93820364,"statfs":{"total":96636764160,"available":96072052736,"internally_reserved":0,"allocated":376090624,"data_stored":375889609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188608207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":7.024},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":7.0579999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.24}]}]},{"osd":1,"up_from":33,"seq":141733920792,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367172,"kb_used_data":195260,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94004668,"statfs":{"total":96636764160,"available":96260780032,"internally_reserved":0,"allocated":199946240,"data_stored":199675645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":176025292},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.20899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.25}]}]},{"osd":0,"up_from":21,"seq":90194313258,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694912,"kb_used_data":428728,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93676928,"statfs":{"total":96636764160,"available":95925174272,"internally_reserved":0,"allocated":439017472,"data_stored":438822919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":272559827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":439975936,"data_stored":439972457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200519680,"data_stored":200446621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376225792,"data_stored":376222313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:54.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 162 20 125517 125497 3.02499 1.92578 0.0194838 0.0258191 2024-04-22T08:29:55.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 163 20 126024 126004 3.01858 1.98047 0.0222023 0.0258739 2024-04-22T08:29:56.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 164 20 126449 126429 3.0103 1.66016 0.0150353 0.0259434 2024-04-22T08:29:57.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:57.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 165 20 126922 126902 3.00326 1.84766 0.0436127 0.0260062 2024-04-22T08:29:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:57.892 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:57.964 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-04-22T08:29:56.196449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513421141,"num_objects":125358,"num_object_clones":0,"num_object_copies":250716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":125358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98960,"num_write_kb":395753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3231,"num_bytes_recovered":13161065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125485,"ondisk_log_size":125485,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655984,"kb_used_data":1011544,"kb_used_omap":12,"kb_used_meta":644403,"kb_avail":281459536,"statfs":{"total":289910292480,"available":288214564864,"internally_reserved":0,"allocated":1035821056,"data_stored":1035154893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":659868782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,7,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29982720,"num_objects":7320,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-576,"num_objects_unfound":0,"num_objects_dirty":7320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":29280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005170"},"pg_stats":[{"pgid":"2.f","version":"35'7931","reported_seq":7991,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.309788+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:54.309788+0000","last_peered":"2024-04-22T08:29:54.309788+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:54.309788+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32485376,"num_objects":7931,"num_object_clones":0,"num_object_copies":15862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":31724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4137},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7905","reported_seq":8018,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.867740+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:54.867740+0000","last_peered":"2024-04-22T08:29:54.867740+0000","last_clean":"2024-04-22T08:29:54.867740+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:54.867740+0000","last_undegraded":"2024-04-22T08:29:54.867740+0000","last_fullsized":"2024-04-22T08:29:54.867740+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7905,"log_dups_size":0,"ondisk_log_size":7905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354304,"num_objects":7899,"num_object_clones":0,"num_object_copies":15798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4961,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7792","reported_seq":7913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.903647+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:54.903647+0000","last_peered":"2024-04-22T08:29:54.903647+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:54.903647+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31899648,"num_objects":7788,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":18476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7869","reported_seq":7896,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.869700+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:54.869700+0000","last_peered":"2024-04-22T08:29:54.869700+0000","last_clean":"2024-04-22T08:29:54.869700+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:54.869700+0000","last_undegraded":"2024-04-22T08:29:54.869700+0000","last_fullsized":"2024-04-22T08:29:54.869700+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32231424,"num_objects":7869,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":31476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7830","reported_seq":7966,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.296736+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:54.296736+0000","last_peered":"2024-04-22T08:29:54.296736+0000","last_clean":"2024-04-22T08:29:54.296736+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:54.296736+0000","last_undegraded":"2024-04-22T08:29:54.296736+0000","last_fullsized":"2024-04-22T08:29:54.296736+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32043008,"num_objects":7823,"num_object_clones":0,"num_object_copies":15646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":19688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7930","reported_seq":8090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.334869+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:54.334869+0000","last_peered":"2024-04-22T08:29:54.334869+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:54.334869+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32460800,"num_objects":7925,"num_object_clones":0,"num_object_copies":15850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":19196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4046},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7741","reported_seq":14323,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:54.345536+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:54.345536+0000","last_peered":"2024-04-22T08:29:54.345536+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:54.345536+0000","last_undegraded":"2024-04-22T08:29:54.345536+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31682560,"num_objects":7735,"num_object_clones":0,"num_object_copies":15470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":7735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":16436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3211,"num_bytes_recovered":13144064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7163},{"shards":"0,2","objects":572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7739","reported_seq":7881,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.346283+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:54.346283+0000","last_peered":"2024-04-22T08:29:54.346283+0000","last_clean":"2024-04-22T08:29:54.346283+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:54.346283+0000","last_undegraded":"2024-04-22T08:29:54.346283+0000","last_fullsized":"2024-04-22T08:29:54.346283+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31690752,"num_objects":7737,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4387,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7970","reported_seq":8091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.854588+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:54.854588+0000","last_peered":"2024-04-22T08:29:54.854588+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:54.854588+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32380114,"num_objects":7907,"num_object_clones":0,"num_object_copies":15814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7910,"num_write_kb":31624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7860","reported_seq":7935,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.292767+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:54.292767+0000","last_peered":"2024-04-22T08:29:54.292767+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:54.292767+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7860,"log_dups_size":0,"ondisk_log_size":7860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32163026,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":31412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4083},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7785","reported_seq":7927,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.826430+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:54.826430+0000","last_peered":"2024-04-22T08:29:54.826430+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:54.826430+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31866880,"num_objects":7780,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":16956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.216998+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.216998+0000","last_peered":"2024-04-22T08:27:56.216998+0000","last_clean":"2024-04-22T08:27:56.216998+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.216998+0000","last_undegraded":"2024-04-22T08:27:56.216998+0000","last_fullsized":"2024-04-22T08:27:56.216998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7833","reported_seq":7886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.820959+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:54.820959+0000","last_peered":"2024-04-22T08:29:54.820959+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:54.820959+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060626,"num_objects":7829,"num_object_clones":0,"num_object_copies":15658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":31311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7816","reported_seq":7892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.225391+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:54.225391+0000","last_peered":"2024-04-22T08:29:54.225391+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:54.225391+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32006761,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":31257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4094},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.353404+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.353404+0000","last_peered":"2024-04-22T08:27:56.353404+0000","last_clean":"2024-04-22T08:27:56.353404+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.353404+0000","last_undegraded":"2024-04-22T08:27:56.353404+0000","last_fullsized":"2024-04-22T08:27:56.353404+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7729","reported_seq":7901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.236158+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:54.236158+0000","last_peered":"2024-04-22T08:29:54.236158+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:54.236158+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31625216,"num_objects":7721,"num_object_clones":0,"num_object_copies":15442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":15840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3884},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7857","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.312390+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:54.312390+0000","last_peered":"2024-04-22T08:29:54.312390+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:54.312390+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32167122,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":31414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4150},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.367049+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.367049+0000","last_peered":"2024-04-22T08:27:56.367049+0000","last_clean":"2024-04-22T08:27:56.367049+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.367049+0000","last_undegraded":"2024-04-22T08:27:56.367049+0000","last_fullsized":"2024-04-22T08:27:56.367049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7898","reported_seq":8039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.333738+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:54.333738+0000","last_peered":"2024-04-22T08:29:54.333738+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:54.333738+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32303524,"num_objects":7890,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":31551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4009},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":513421141,"num_objects":125358,"num_object_clones":0,"num_object_copies":250716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":125358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98960,"num_write_kb":395753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3231,"num_bytes_recovered":13161065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1037447168,"data_stored":1037367151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125485,"ondisk_log_size":125485,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561928,"kb_used_data":371968,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93809912,"statfs":{"total":96636764160,"available":96061349888,"internally_reserved":0,"allocated":380895232,"data_stored":380694217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194506447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":4.8470000000000004},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":4.8789999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":5.173},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.35699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920793,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383700,"kb_used_data":203596,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93988140,"statfs":{"total":96636764160,"available":96243855360,"internally_reserved":0,"allocated":208482304,"data_stored":208211709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":184413900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":2.6859999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.627}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":7.7300000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313259,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710356,"kb_used_data":435980,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93661484,"statfs":{"total":96636764160,"available":95909359616,"internally_reserved":0,"allocated":446443520,"data_stored":446248967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":280948435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":447356928,"data_stored":447353449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":209027072,"data_stored":208954013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381063168,"data_stored":381059689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:57.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:29:58.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:29:58.840 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:29:58.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 166 20 127385 127365 2.99606 1.80859 0.0404509 0.026068 2024-04-22T08:29:58.920 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-04-22T08:29:58.200392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513421141,"num_objects":125358,"num_object_clones":0,"num_object_copies":250716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":125358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98960,"num_write_kb":395753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3231,"num_bytes_recovered":13161065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125485,"ondisk_log_size":125485,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1673696,"kb_used_data":1019400,"kb_used_omap":12,"kb_used_meta":654259,"kb_avail":281441824,"statfs":{"total":289910292480,"available":288196427776,"internally_reserved":0,"allocated":1043865600,"data_stored":1043199437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":669961326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,5,14,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19595264,"num_objects":4784,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":19136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008681"},"pg_stats":[{"pgid":"2.f","version":"35'7931","reported_seq":7991,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.309788+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:54.309788+0000","last_peered":"2024-04-22T08:29:54.309788+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:54.309788+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32485376,"num_objects":7931,"num_object_clones":0,"num_object_copies":15862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":31724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4137},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'7905","reported_seq":8018,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.867740+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:54.867740+0000","last_peered":"2024-04-22T08:29:54.867740+0000","last_clean":"2024-04-22T08:29:54.867740+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:54.867740+0000","last_undegraded":"2024-04-22T08:29:54.867740+0000","last_fullsized":"2024-04-22T08:29:54.867740+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":7905,"log_dups_size":0,"ondisk_log_size":7905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354304,"num_objects":7899,"num_object_clones":0,"num_object_copies":15798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4961,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7792","reported_seq":7913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.903647+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:54.903647+0000","last_peered":"2024-04-22T08:29:54.903647+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:54.903647+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31899648,"num_objects":7788,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":18476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'7869","reported_seq":7896,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.869700+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:54.869700+0000","last_peered":"2024-04-22T08:29:54.869700+0000","last_clean":"2024-04-22T08:29:54.869700+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:54.869700+0000","last_undegraded":"2024-04-22T08:29:54.869700+0000","last_fullsized":"2024-04-22T08:29:54.869700+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32231424,"num_objects":7869,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":31476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7830","reported_seq":7966,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.296736+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:54.296736+0000","last_peered":"2024-04-22T08:29:54.296736+0000","last_clean":"2024-04-22T08:29:54.296736+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:54.296736+0000","last_undegraded":"2024-04-22T08:29:54.296736+0000","last_fullsized":"2024-04-22T08:29:54.296736+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32043008,"num_objects":7823,"num_object_clones":0,"num_object_copies":15646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":19688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'7930","reported_seq":8090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.334869+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:54.334869+0000","last_peered":"2024-04-22T08:29:54.334869+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:54.334869+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32460800,"num_objects":7925,"num_object_clones":0,"num_object_copies":15850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":19196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4046},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7741","reported_seq":14323,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:54.345536+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:54.345536+0000","last_peered":"2024-04-22T08:29:54.345536+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:54.345536+0000","last_undegraded":"2024-04-22T08:29:54.345536+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31682560,"num_objects":7735,"num_object_clones":0,"num_object_copies":15470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":7735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":16436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3211,"num_bytes_recovered":13144064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7163},{"shards":"0,2","objects":572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7739","reported_seq":7881,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:54.346283+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:54.346283+0000","last_peered":"2024-04-22T08:29:54.346283+0000","last_clean":"2024-04-22T08:29:54.346283+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:54.346283+0000","last_undegraded":"2024-04-22T08:29:54.346283+0000","last_fullsized":"2024-04-22T08:29:54.346283+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31690752,"num_objects":7737,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4387,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'7970","reported_seq":8091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.854588+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:54.854588+0000","last_peered":"2024-04-22T08:29:54.854588+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:54.854588+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32380114,"num_objects":7907,"num_object_clones":0,"num_object_copies":15814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7910,"num_write_kb":31624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'7860","reported_seq":7935,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.292767+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:54.292767+0000","last_peered":"2024-04-22T08:29:54.292767+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:54.292767+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7860,"log_dups_size":0,"ondisk_log_size":7860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32163026,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":31412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4083},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7785","reported_seq":7927,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.826430+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:54.826430+0000","last_peered":"2024-04-22T08:29:54.826430+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:54.826430+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7785,"log_dups_size":0,"ondisk_log_size":7785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31866880,"num_objects":7780,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":16956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7833","reported_seq":7886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.820959+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:54.820959+0000","last_peered":"2024-04-22T08:29:54.820959+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:54.820959+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060626,"num_objects":7829,"num_object_clones":0,"num_object_copies":15658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":31311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7816","reported_seq":7892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.225391+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:54.225391+0000","last_peered":"2024-04-22T08:29:54.225391+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:54.225391+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32006761,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":31257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4094},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7729","reported_seq":7901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.236158+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:54.236158+0000","last_peered":"2024-04-22T08:29:54.236158+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:54.236158+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31625216,"num_objects":7721,"num_object_clones":0,"num_object_copies":15442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":15840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3884},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'7857","reported_seq":7931,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.312390+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:54.312390+0000","last_peered":"2024-04-22T08:29:54.312390+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:54.312390+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32167122,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":31414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4150},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'7898","reported_seq":8039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:54.333738+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:54.333738+0000","last_peered":"2024-04-22T08:29:54.333738+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:54.333738+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32303524,"num_objects":7890,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7895,"num_write_kb":31551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4009},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":513421141,"num_objects":125358,"num_object_clones":0,"num_object_copies":250716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":572,"num_objects_unfound":0,"num_objects_dirty":125358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98960,"num_write_kb":395753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3231,"num_bytes_recovered":13161065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045528576,"data_stored":1045448559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125485,"ondisk_log_size":125485,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561928,"kb_used_data":371968,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93809912,"statfs":{"total":96636764160,"available":96061349888,"internally_reserved":0,"allocated":380895232,"data_stored":380694217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194506447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":4.8470000000000004},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":4.8789999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":5.173},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":0.35699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920794,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401412,"kb_used_data":211452,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93970428,"statfs":{"total":96636764160,"available":96225718272,"internally_reserved":0,"allocated":216526848,"data_stored":216256253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":194506444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.6499999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.23300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.311},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.253}]}]},{"osd":0,"up_from":21,"seq":90194313259,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710356,"kb_used_data":435980,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93661484,"statfs":{"total":96636764160,"available":95909359616,"internally_reserved":0,"allocated":446443520,"data_stored":446248967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":280948435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.258,"5min":5.258,"15min":5.258},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":447356928,"data_stored":447353449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217108480,"data_stored":217035421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381063168,"data_stored":381059689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:29:58.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:29:58.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'7931", 'reported_seq': 7991, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.309788+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:54.309788+0000', 'last_peered': '2024-04-22T08:29:54.309788+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:54.309788+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7931, 'log_dups_size': 0, 'ondisk_log_size': 7931, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32485376, 'num_objects': 7931, 'num_object_clones': 0, 'num_object_copies': 15862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7931, 'num_write_kb': 31724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4137}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:29:58.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7792", 'reported_seq': 7913, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.903647+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:54.903647+0000', 'last_peered': '2024-04-22T08:29:54.903647+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:54.903647+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7792, 'log_dups_size': 0, 'ondisk_log_size': 7792, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31899648, 'num_objects': 7788, 'num_object_clones': 0, 'num_object_copies': 15576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4619, 'num_write_kb': 18476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 4033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'7930", 'reported_seq': 8090, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.334869+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:54.334869+0000', 'last_peered': '2024-04-22T08:29:54.334869+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:54.334869+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 7930, 'log_dups_size': 0, 'ondisk_log_size': 7930, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32460800, 'num_objects': 7925, 'num_object_clones': 0, 'num_object_copies': 15850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4799, 'num_write_kb': 19196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4046}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'7741", 'reported_seq': 14323, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:54.345536+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:54.345536+0000', 'last_peered': '2024-04-22T08:29:54.345536+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:54.345536+0000', 'last_undegraded': '2024-04-22T08:29:54.345536+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 7741, 'log_dups_size': 0, 'ondisk_log_size': 7741, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31682560, 'num_objects': 7735, 'num_object_clones': 0, 'num_object_copies': 15470, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 572, 'num_objects_unfound': 0, 'num_objects_dirty': 7735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4109, 'num_write_kb': 16436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3211, 'num_bytes_recovered': 13144064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7163}, {'shards': '0,2', 'objects': 572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'7970", 'reported_seq': 8091, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.854588+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:54.854588+0000', 'last_peered': '2024-04-22T08:29:54.854588+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:54.854588+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 7970, 'log_dups_size': 0, 'ondisk_log_size': 7970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32380114, 'num_objects': 7907, 'num_object_clones': 0, 'num_object_copies': 15814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7910, 'num_write_kb': 31624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 4019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:29:58.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'7860", 'reported_seq': 7935, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.292767+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:54.292767+0000', 'last_peered': '2024-04-22T08:29:54.292767+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:54.292767+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7860, 'log_dups_size': 0, 'ondisk_log_size': 7860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32163026, 'num_objects': 7854, 'num_object_clones': 0, 'num_object_copies': 15708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7858, 'num_write_kb': 31412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4083}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:29:58.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7785", 'reported_seq': 7927, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.826430+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:54.826430+0000', 'last_peered': '2024-04-22T08:29:54.826430+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:54.826430+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7785, 'log_dups_size': 0, 'ondisk_log_size': 7785, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31866880, 'num_objects': 7780, 'num_object_clones': 0, 'num_object_copies': 15560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4239, 'num_write_kb': 16956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 4030}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:58.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:29:58.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7833", 'reported_seq': 7886, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.820959+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:54.820959+0000', 'last_peered': '2024-04-22T08:29:54.820959+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:54.820959+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7833, 'log_dups_size': 0, 'ondisk_log_size': 7833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32060626, 'num_objects': 7829, 'num_object_clones': 0, 'num_object_copies': 15658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7831, 'num_write_kb': 31311, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 4026}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:29:58.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:29:58.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7816", 'reported_seq': 7892, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.225391+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:54.225391+0000', 'last_peered': '2024-04-22T08:29:54.225391+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:54.225391+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7816, 'log_dups_size': 0, 'ondisk_log_size': 7816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32006761, 'num_objects': 7815, 'num_object_clones': 0, 'num_object_copies': 15630, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7815, 'num_write_kb': 31257, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4094}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'7729", 'reported_seq': 7901, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.236158+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:54.236158+0000', 'last_peered': '2024-04-22T08:29:54.236158+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:54.236158+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 7729, 'log_dups_size': 0, 'ondisk_log_size': 7729, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31625216, 'num_objects': 7721, 'num_object_clones': 0, 'num_object_copies': 15442, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3960, 'num_write_kb': 15840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3884}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'7857", 'reported_seq': 7931, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.312390+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:54.312390+0000', 'last_peered': '2024-04-22T08:29:54.312390+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:54.312390+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7857, 'log_dups_size': 0, 'ondisk_log_size': 7857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32167122, 'num_objects': 7855, 'num_object_clones': 0, 'num_object_copies': 15710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7855, 'num_write_kb': 31414, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4150}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:29:58.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'7898", 'reported_seq': 8039, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:54.333738+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:54.333738+0000', 'last_peered': '2024-04-22T08:29:54.333738+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:54.333738+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 7898, 'log_dups_size': 0, 'ondisk_log_size': 7898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32303524, 'num_objects': 7890, 'num_object_clones': 0, 'num_object_copies': 15780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7895, 'num_write_kb': 31551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4009}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:29:58.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:29:59.870 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 167 20 127862 127842 2.98928 1.86328 0.0433951 0.026126 2024-04-22T08:30:00.016 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:00.016 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/250716 objects degraded (16.674%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":234,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":125358,"data_bytes":513421141,"bytes_used":1713864704,"bytes_avail":288196427776,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":250716,"degraded_ratio":0.16673846104755982,"misplaced_objects":572,"misplaced_total":250716,"misplaced_ratio":0.0022814658817147688,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":135070,"recovering_keys_per_sec":0,"num_objects_recovered":396,"num_bytes_recovered":1622016,"num_keys_recovered":0,"write_bytes_sec":1631758,"read_op_per_sec":0,"write_op_per_sec":398},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-22T08:28:52.169443+0000","services":{}},"progress_events":{}} 2024-04-22T08:30:00.057 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:00.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:00.872 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 168 20 128327 128307 2.98226 1.81641 0.0291984 0.0261882 2024-04-22T08:30:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:00.955 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:01.022 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-04-22T08:30:00.202593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1677492,"kb_used_data":1031388,"kb_used_omap":12,"kb_used_meta":646067,"kb_avail":281438028,"statfs":{"total":289910292480,"available":288192540672,"internally_reserved":0,"allocated":1056141312,"data_stored":1055475149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":661572718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,15,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29360128,"num_objects":7168,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-577,"num_objects_unfound":0,"num_objects_dirty":7168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7168,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":2363392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009595"},"pg_stats":[{"pgid":"2.f","version":"35'8077","reported_seq":8138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.276422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:59.276422+0000","last_peered":"2024-04-22T08:29:59.276422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:59.276422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8077,"num_write_kb":32308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4283},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'8052","reported_seq":8165,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.814818+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:59.814818+0000","last_peered":"2024-04-22T08:29:59.814818+0000","last_clean":"2024-04-22T08:29:59.814818+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:59.814818+0000","last_undegraded":"2024-04-22T08:29:59.814818+0000","last_fullsized":"2024-04-22T08:29:59.814818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32956416,"num_objects":8046,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":20432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7928","reported_seq":8049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.840714+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:59.840714+0000","last_peered":"2024-04-22T08:29:59.840714+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:59.840714+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32456704,"num_objects":7924,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":19020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'8028","reported_seq":8056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.813968+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:59.813968+0000","last_peered":"2024-04-22T08:29:59.813968+0000","last_clean":"2024-04-22T08:29:59.813968+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:59.813968+0000","last_undegraded":"2024-04-22T08:29:59.813968+0000","last_fullsized":"2024-04-22T08:29:59.813968+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32882688,"num_objects":8028,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7971","reported_seq":8106,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.388246+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:59.388246+0000","last_peered":"2024-04-22T08:29:59.388246+0000","last_clean":"2024-04-22T08:29:59.388246+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:59.388246+0000","last_undegraded":"2024-04-22T08:29:59.388246+0000","last_fullsized":"2024-04-22T08:29:59.388246+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":20252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'8082","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.388255+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:59.388255+0000","last_peered":"2024-04-22T08:29:59.388255+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:59.388255+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":19804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4198},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7896","reported_seq":14840,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:59.393053+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:59.393053+0000","last_peered":"2024-04-22T08:29:59.393053+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:59.393053+0000","last_undegraded":"2024-04-22T08:29:59.393053+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32317440,"num_objects":7890,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":7890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4264,"num_write_kb":17056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3392,"num_bytes_recovered":13885440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7499},{"shards":"0,2","objects":391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7903","reported_seq":8045,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.331925+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:59.331925+0000","last_peered":"2024-04-22T08:29:59.331925+0000","last_clean":"2024-04-22T08:29:59.331925+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:59.331925+0000","last_undegraded":"2024-04-22T08:29:59.331925+0000","last_fullsized":"2024-04-22T08:29:59.331925+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32362496,"num_objects":7901,"num_object_clones":0,"num_object_copies":15802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":18204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'8108","reported_seq":8229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.643082+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:59.643082+0000","last_peered":"2024-04-22T08:29:59.643082+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:59.643082+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32945362,"num_objects":8045,"num_object_clones":0,"num_object_copies":16090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8048,"num_write_kb":32176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'8005","reported_seq":8080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.386106+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:59.386106+0000","last_peered":"2024-04-22T08:29:59.386106+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:59.386106+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756946,"num_objects":7999,"num_object_clones":0,"num_object_copies":15998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":31992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4228},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7943","reported_seq":8087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.829263+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:59.829263+0000","last_peered":"2024-04-22T08:29:59.829263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:59.829263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32514048,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4188}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7989","reported_seq":8035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.839094+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:59.839094+0000","last_peered":"2024-04-22T08:29:59.839094+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:59.839094+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699602,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":31935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7970","reported_seq":8047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.367064+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:59.367064+0000","last_peered":"2024-04-22T08:29:59.367064+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:59.367064+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32637545,"num_objects":7969,"num_object_clones":0,"num_object_copies":15938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":31873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4248},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7862","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.377183+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:59.377183+0000","last_peered":"2024-04-22T08:29:59.377183+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:59.377183+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32169984,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":16372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4017},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'8012","reported_seq":8086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.216828+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:59.216828+0000","last_peered":"2024-04-22T08:29:59.216828+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:59.216828+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32802002,"num_objects":8010,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":32034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.102298+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.102298+0000","last_peered":"2024-04-22T08:27:56.102298+0000","last_clean":"2024-04-22T08:27:56.102298+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.102298+0000","last_undegraded":"2024-04-22T08:27:56.102298+0000","last_fullsized":"2024-04-22T08:27:56.102298+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'8043","reported_seq":8183,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.357061+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:59.357061+0000","last_peered":"2024-04-22T08:29:59.357061+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:59.357061+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897444,"num_objects":8035,"num_object_clones":0,"num_object_copies":16070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":32131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057746944,"data_stored":1057666927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117032,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570864,"kb_used_data":376808,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93800976,"statfs":{"total":96636764160,"available":96052199424,"internally_reserved":0,"allocated":385851392,"data_stored":385650377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.8169999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.24099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920794,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401412,"kb_used_data":211452,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93970428,"statfs":{"total":96636764160,"available":96225718272,"internally_reserved":0,"allocated":216526848,"data_stored":216256253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":194506444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.6499999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.23300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.6379999999999999,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.311},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.2969999999999997,"5min":6.2969999999999997,"15min":6.2969999999999997},"last":0.253}]}]},{"osd":0,"up_from":21,"seq":90194313260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705216,"kb_used_data":443128,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93666624,"statfs":{"total":96636764160,"available":95914622976,"internally_reserved":0,"allocated":453763072,"data_stored":453568519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":268365523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":454696960,"data_stored":454693481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217108480,"data_stored":217035421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385941504,"data_stored":385938025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:01.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 169 20 128785 128765 2.9752 1.78906 0.0297333 0.0262513 2024-04-22T08:30:02.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 170 20 129249 129229 2.96836 1.8125 0.0424305 0.0263126 2024-04-22T08:30:03.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 171 20 129754 129734 2.96254 1.97266 0.0328873 0.0263648 2024-04-22T08:30:04.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:04.713 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:04.714 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:04.782 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-04-22T08:30:04.204075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711512,"kb_used_data":1039168,"kb_used_omap":12,"kb_used_meta":672307,"kb_avail":281404008,"statfs":{"total":289910292480,"available":288157704192,"internally_reserved":0,"allocated":1064108032,"data_stored":1063441869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":688442478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,2,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19640320,"num_objects":4795,"num_object_clones":0,"num_object_copies":9590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-391,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4795,"num_write_kb":19180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":1601536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008612"},"pg_stats":[{"pgid":"2.f","version":"35'8077","reported_seq":8138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.276422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:59.276422+0000","last_peered":"2024-04-22T08:29:59.276422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:59.276422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8077,"num_write_kb":32308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4283},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'8052","reported_seq":8165,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.814818+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:59.814818+0000","last_peered":"2024-04-22T08:29:59.814818+0000","last_clean":"2024-04-22T08:29:59.814818+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:59.814818+0000","last_undegraded":"2024-04-22T08:29:59.814818+0000","last_fullsized":"2024-04-22T08:29:59.814818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32956416,"num_objects":8046,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":20432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7928","reported_seq":8049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.840714+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:59.840714+0000","last_peered":"2024-04-22T08:29:59.840714+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:59.840714+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32456704,"num_objects":7924,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":19020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'8028","reported_seq":8056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.813968+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:59.813968+0000","last_peered":"2024-04-22T08:29:59.813968+0000","last_clean":"2024-04-22T08:29:59.813968+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:59.813968+0000","last_undegraded":"2024-04-22T08:29:59.813968+0000","last_fullsized":"2024-04-22T08:29:59.813968+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32882688,"num_objects":8028,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7971","reported_seq":8106,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.388246+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:59.388246+0000","last_peered":"2024-04-22T08:29:59.388246+0000","last_clean":"2024-04-22T08:29:59.388246+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:59.388246+0000","last_undegraded":"2024-04-22T08:29:59.388246+0000","last_fullsized":"2024-04-22T08:29:59.388246+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":20252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'8082","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.388255+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:59.388255+0000","last_peered":"2024-04-22T08:29:59.388255+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:59.388255+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":19804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4198},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7896","reported_seq":14840,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:59.393053+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:59.393053+0000","last_peered":"2024-04-22T08:29:59.393053+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:59.393053+0000","last_undegraded":"2024-04-22T08:29:59.393053+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32317440,"num_objects":7890,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":7890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4264,"num_write_kb":17056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3392,"num_bytes_recovered":13885440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7499},{"shards":"0,2","objects":391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7903","reported_seq":8045,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.331925+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:59.331925+0000","last_peered":"2024-04-22T08:29:59.331925+0000","last_clean":"2024-04-22T08:29:59.331925+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:59.331925+0000","last_undegraded":"2024-04-22T08:29:59.331925+0000","last_fullsized":"2024-04-22T08:29:59.331925+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32362496,"num_objects":7901,"num_object_clones":0,"num_object_copies":15802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":18204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'8108","reported_seq":8229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.643082+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:59.643082+0000","last_peered":"2024-04-22T08:29:59.643082+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:59.643082+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32945362,"num_objects":8045,"num_object_clones":0,"num_object_copies":16090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8048,"num_write_kb":32176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'8005","reported_seq":8080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.386106+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:59.386106+0000","last_peered":"2024-04-22T08:29:59.386106+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:59.386106+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756946,"num_objects":7999,"num_object_clones":0,"num_object_copies":15998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":31992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4228},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7943","reported_seq":8087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.829263+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:59.829263+0000","last_peered":"2024-04-22T08:29:59.829263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:59.829263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32514048,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4188}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7989","reported_seq":8035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.839094+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:59.839094+0000","last_peered":"2024-04-22T08:29:59.839094+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:59.839094+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699602,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":31935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7970","reported_seq":8047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.367064+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:59.367064+0000","last_peered":"2024-04-22T08:29:59.367064+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:59.367064+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32637545,"num_objects":7969,"num_object_clones":0,"num_object_copies":15938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":31873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4248},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7862","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.377183+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:59.377183+0000","last_peered":"2024-04-22T08:29:59.377183+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:59.377183+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32169984,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":16372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4017},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'8012","reported_seq":8086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.216828+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:59.216828+0000","last_peered":"2024-04-22T08:29:59.216828+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:59.216828+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32802002,"num_objects":8010,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":32034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.104008+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.104008+0000","last_peered":"2024-04-22T08:27:56.104008+0000","last_clean":"2024-04-22T08:27:56.104008+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.104008+0000","last_undegraded":"2024-04-22T08:27:56.104008+0000","last_fullsized":"2024-04-22T08:27:56.104008+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'8043","reported_seq":8183,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.357061+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:59.357061+0000","last_peered":"2024-04-22T08:29:59.357061+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:59.357061+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897444,"num_objects":8035,"num_object_clones":0,"num_object_copies":16070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":32131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1065832448,"data_stored":1065752431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117032,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570864,"kb_used_data":376808,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93800976,"statfs":{"total":96636764160,"available":96052199424,"internally_reserved":0,"allocated":385851392,"data_stored":385650377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.8169999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.24099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920795,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435432,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93936408,"statfs":{"total":96636764160,"available":96190881792,"internally_reserved":0,"allocated":224493568,"data_stored":224222973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.0149999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.29499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705216,"kb_used_data":443128,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93666624,"statfs":{"total":96636764160,"available":95914622976,"internally_reserved":0,"allocated":453763072,"data_stored":453568519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":268365523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":454696960,"data_stored":454693481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225193984,"data_stored":225120925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385941504,"data_stored":385938025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:04.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:04.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 172 20 130207 130187 2.9556 1.76953 0.0496172 0.0264243 2024-04-22T08:30:05.469 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:05.471 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:05.535 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-04-22T08:30:04.204075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711512,"kb_used_data":1039168,"kb_used_omap":12,"kb_used_meta":672307,"kb_avail":281404008,"statfs":{"total":289910292480,"available":288157704192,"internally_reserved":0,"allocated":1064108032,"data_stored":1063441869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":688442478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,2,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19640320,"num_objects":4795,"num_object_clones":0,"num_object_copies":9590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-391,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4795,"num_write_kb":19180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":1601536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008612"},"pg_stats":[{"pgid":"2.f","version":"35'8077","reported_seq":8138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.276422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:29:59.276422+0000","last_peered":"2024-04-22T08:29:59.276422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:29:59.276422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8077,"num_write_kb":32308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4283},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'8052","reported_seq":8165,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.814818+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:29:59.814818+0000","last_peered":"2024-04-22T08:29:59.814818+0000","last_clean":"2024-04-22T08:29:59.814818+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:29:59.814818+0000","last_undegraded":"2024-04-22T08:29:59.814818+0000","last_fullsized":"2024-04-22T08:29:59.814818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32956416,"num_objects":8046,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":20432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'7928","reported_seq":8049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.840714+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:29:59.840714+0000","last_peered":"2024-04-22T08:29:59.840714+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:29:59.840714+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32456704,"num_objects":7924,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":19020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'8028","reported_seq":8056,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.813968+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:29:59.813968+0000","last_peered":"2024-04-22T08:29:59.813968+0000","last_clean":"2024-04-22T08:29:59.813968+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:29:59.813968+0000","last_undegraded":"2024-04-22T08:29:59.813968+0000","last_fullsized":"2024-04-22T08:29:59.813968+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32882688,"num_objects":8028,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'7971","reported_seq":8106,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.388246+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:29:59.388246+0000","last_peered":"2024-04-22T08:29:59.388246+0000","last_clean":"2024-04-22T08:29:59.388246+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:29:59.388246+0000","last_undegraded":"2024-04-22T08:29:59.388246+0000","last_fullsized":"2024-04-22T08:29:59.388246+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":20252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'8082","reported_seq":8241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.388255+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:29:59.388255+0000","last_peered":"2024-04-22T08:29:59.388255+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:29:59.388255+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083392,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":19804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4198},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'7896","reported_seq":14840,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:29:59.393053+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:29:59.393053+0000","last_peered":"2024-04-22T08:29:59.393053+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:29:59.393053+0000","last_undegraded":"2024-04-22T08:29:59.393053+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32317440,"num_objects":7890,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":7890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4264,"num_write_kb":17056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3392,"num_bytes_recovered":13885440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7499},{"shards":"0,2","objects":391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'7903","reported_seq":8045,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:59.331925+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:29:59.331925+0000","last_peered":"2024-04-22T08:29:59.331925+0000","last_clean":"2024-04-22T08:29:59.331925+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:29:59.331925+0000","last_undegraded":"2024-04-22T08:29:59.331925+0000","last_fullsized":"2024-04-22T08:29:59.331925+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32362496,"num_objects":7901,"num_object_clones":0,"num_object_copies":15802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":18204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'8108","reported_seq":8229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.643082+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:29:59.643082+0000","last_peered":"2024-04-22T08:29:59.643082+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:29:59.643082+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32945362,"num_objects":8045,"num_object_clones":0,"num_object_copies":16090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8048,"num_write_kb":32176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'8005","reported_seq":8080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.386106+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:29:59.386106+0000","last_peered":"2024-04-22T08:29:59.386106+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:29:59.386106+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756946,"num_objects":7999,"num_object_clones":0,"num_object_copies":15998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":31992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4228},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'7943","reported_seq":8087,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.829263+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:29:59.829263+0000","last_peered":"2024-04-22T08:29:59.829263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:29:59.829263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32514048,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":17588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4188}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'7989","reported_seq":8035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.839094+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:29:59.839094+0000","last_peered":"2024-04-22T08:29:59.839094+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:29:59.839094+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699602,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":31935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'7970","reported_seq":8047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.367064+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:29:59.367064+0000","last_peered":"2024-04-22T08:29:59.367064+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:29:59.367064+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32637545,"num_objects":7969,"num_object_clones":0,"num_object_copies":15938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":31873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4248},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'7862","reported_seq":8034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.377183+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:29:59.377183+0000","last_peered":"2024-04-22T08:29:59.377183+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:29:59.377183+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32169984,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":16372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4017},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'8012","reported_seq":8086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.216828+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:29:59.216828+0000","last_peered":"2024-04-22T08:29:59.216828+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:29:59.216828+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32802002,"num_objects":8010,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":32034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.104008+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.104008+0000","last_peered":"2024-04-22T08:27:56.104008+0000","last_clean":"2024-04-22T08:27:56.104008+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.104008+0000","last_undegraded":"2024-04-22T08:27:56.104008+0000","last_fullsized":"2024-04-22T08:27:56.104008+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'8043","reported_seq":8183,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:29:59.357061+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:29:59.357061+0000","last_peered":"2024-04-22T08:29:59.357061+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:29:59.357061+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32897444,"num_objects":8035,"num_object_clones":0,"num_object_copies":16070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":32131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523186005,"num_objects":127742,"num_object_clones":0,"num_object_copies":255484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":391,"num_objects_unfound":0,"num_objects_dirty":127742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101344,"num_write_kb":405289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":13902441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1065832448,"data_stored":1065752431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127869,"ondisk_log_size":127869,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117032,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570864,"kb_used_data":376808,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93800976,"statfs":{"total":96636764160,"available":96052199424,"internally_reserved":0,"allocated":385851392,"data_stored":385650377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":1.8169999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.24099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920795,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435432,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93936408,"statfs":{"total":96636764160,"available":96190881792,"internally_reserved":0,"allocated":224493568,"data_stored":224222973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.0149999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.29499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705216,"kb_used_data":443128,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93666624,"statfs":{"total":96636764160,"available":95914622976,"internally_reserved":0,"allocated":453763072,"data_stored":453568519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":268365523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":454696960,"data_stored":454693481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225193984,"data_stored":225120925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385941504,"data_stored":385938025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "35'8077", 'reported_seq': 8138, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.276422+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:29:59.276422+0000', 'last_peered': '2024-04-22T08:29:59.276422+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:29:59.276422+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8077, 'log_dups_size': 0, 'ondisk_log_size': 8077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33083392, 'num_objects': 8077, 'num_object_clones': 0, 'num_object_copies': 16154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8077, 'num_write_kb': 32308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4283}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "35'7928", 'reported_seq': 8049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.840714+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:29:59.840714+0000', 'last_peered': '2024-04-22T08:29:59.840714+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:29:59.840714+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 7928, 'log_dups_size': 0, 'ondisk_log_size': 7928, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32456704, 'num_objects': 7924, 'num_object_clones': 0, 'num_object_copies': 15848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4755, 'num_write_kb': 19020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 4169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "35'8082", 'reported_seq': 8241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.388255+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:29:59.388255+0000', 'last_peered': '2024-04-22T08:29:59.388255+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:29:59.388255+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 8082, 'log_dups_size': 0, 'ondisk_log_size': 8082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33083392, 'num_objects': 8077, 'num_object_clones': 0, 'num_object_copies': 16154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4951, 'num_write_kb': 19804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4198}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:30:05.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'7896", 'reported_seq': 14840, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:29:59.393053+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:29:59.393053+0000', 'last_peered': '2024-04-22T08:29:59.393053+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:29:59.393053+0000', 'last_undegraded': '2024-04-22T08:29:59.393053+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 7896, 'log_dups_size': 0, 'ondisk_log_size': 7896, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32317440, 'num_objects': 7890, 'num_object_clones': 0, 'num_object_copies': 15780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 391, 'num_objects_unfound': 0, 'num_objects_dirty': 7890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4264, 'num_write_kb': 17056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3392, 'num_bytes_recovered': 13885440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7499}, {'shards': '0,2', 'objects': 391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "35'8108", 'reported_seq': 8229, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.643082+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:29:59.643082+0000', 'last_peered': '2024-04-22T08:29:59.643082+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:29:59.643082+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 8108, 'log_dups_size': 0, 'ondisk_log_size': 8108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32945362, 'num_objects': 8045, 'num_object_clones': 0, 'num_object_copies': 16090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8048, 'num_write_kb': 32176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 4157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "35'8005", 'reported_seq': 8080, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.386106+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:29:59.386106+0000', 'last_peered': '2024-04-22T08:29:59.386106+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:29:59.386106+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8005, 'log_dups_size': 0, 'ondisk_log_size': 8005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32756946, 'num_objects': 7999, 'num_object_clones': 0, 'num_object_copies': 15998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8003, 'num_write_kb': 31992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4228}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "35'7943", 'reported_seq': 8087, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.829263+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:29:59.829263+0000', 'last_peered': '2024-04-22T08:29:59.829263+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:29:59.829263+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32514048, 'num_objects': 7938, 'num_object_clones': 0, 'num_object_copies': 15876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4397, 'num_write_kb': 17588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 4188}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "35'7989", 'reported_seq': 8035, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.839094+0000', 'last_change': '2024-04-22T08:27:56.632687+0000', 'last_active': '2024-04-22T08:29:59.839094+0000', 'last_peered': '2024-04-22T08:29:59.839094+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:29:59.839094+0000', 'last_undegraded': '2024-04-22T08:27:56.180186+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7989, 'log_dups_size': 0, 'ondisk_log_size': 7989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699602, 'num_objects': 7985, 'num_object_clones': 0, 'num_object_copies': 15970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7987, 'num_write_kb': 31935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3803}, {'shards': '2', 'objects': 4182}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:05.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "35'7970", 'reported_seq': 8047, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.367064+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:29:59.367064+0000', 'last_peered': '2024-04-22T08:29:59.367064+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:29:59.367064+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 7970, 'log_dups_size': 0, 'ondisk_log_size': 7970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32637545, 'num_objects': 7969, 'num_object_clones': 0, 'num_object_copies': 15938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7969, 'num_write_kb': 31873, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4248}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "35'7862", 'reported_seq': 8034, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.377183+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:29:59.377183+0000', 'last_peered': '2024-04-22T08:29:59.377183+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:29:59.377183+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 7862, 'log_dups_size': 0, 'ondisk_log_size': 7862, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32169984, 'num_objects': 7854, 'num_object_clones': 0, 'num_object_copies': 15708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4093, 'num_write_kb': 16372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4017}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "35'8012", 'reported_seq': 8086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.216828+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:29:59.216828+0000', 'last_peered': '2024-04-22T08:29:59.216828+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:29:59.216828+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8012, 'log_dups_size': 0, 'ondisk_log_size': 8012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32802002, 'num_objects': 8010, 'num_object_clones': 0, 'num_object_copies': 16020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8010, 'num_write_kb': 32034, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4305}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:05.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "35'8043", 'reported_seq': 8183, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:29:59.357061+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:29:59.357061+0000', 'last_peered': '2024-04-22T08:29:59.357061+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:29:59.357061+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 8043, 'log_dups_size': 0, 'ondisk_log_size': 8043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32897444, 'num_objects': 8035, 'num_object_clones': 0, 'num_object_copies': 16070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8040, 'num_write_kb': 32131, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4154}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:05.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:05.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 173 20 130653 130633 2.94859 1.74219 0.0068527 0.0264875 2024-04-22T08:30:06.448 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:06.448 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41804/260182 objects degraded (16.067%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":241,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":130091,"data_bytes":532807509,"bytes_used":1795493888,"bytes_avail":288114798592,"bytes_total":289910292480,"degraded_objects":41804,"degraded_total":260182,"degraded_ratio":0.16067214488319714,"misplaced_objects":199,"misplaced_total":260182,"misplaced_ratio":0.00076484922092996438,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":198854,"recovering_keys_per_sec":0,"num_objects_recovered":583,"num_bytes_recovered":2387968,"num_keys_recovered":0,"write_bytes_sec":2436737,"read_op_per_sec":0,"write_op_per_sec":594},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-22T08:30:02.203547+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:30:06.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:06.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:06.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 174 20 131114 131094 2.94199 1.80078 0.0217422 0.0265467 2024-04-22T08:30:07.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:07.483 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:07.545 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-22T08:30:06.204579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532807509,"num_objects":130091,"num_object_clones":0,"num_object_copies":260182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":199,"num_objects_unfound":0,"num_objects_dirty":130091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103693,"num_write_kb":414685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3604,"num_bytes_recovered":14688873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130218,"ondisk_log_size":130218,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1753412,"kb_used_data":1050732,"kb_used_omap":12,"kb_used_meta":702643,"kb_avail":281362108,"statfs":{"total":289910292480,"available":288114798592,"internally_reserved":0,"allocated":1075949568,"data_stored":1075283405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":719506542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,6,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29261824,"num_objects":7144,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-583,"num_objects_unfound":0,"num_objects_dirty":7144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":28576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":2387968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008605"},"pg_stats":[{"pgid":"2.f","version":"35'8228","reported_seq":8284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.403583+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:04.403583+0000","last_peered":"2024-04-22T08:30:04.403583+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:04.403583+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8228,"log_dups_size":0,"ondisk_log_size":8228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33701888,"num_objects":8228,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":32912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4434},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'8213","reported_seq":8326,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:30:04.916495+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:04.916495+0000","last_peered":"2024-04-22T08:30:04.916495+0000","last_clean":"2024-04-22T08:30:04.916495+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:04.916495+0000","last_undegraded":"2024-04-22T08:30:04.916495+0000","last_fullsized":"2024-04-22T08:30:04.916495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33615872,"num_objects":8207,"num_object_clones":0,"num_object_copies":16414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'8069","reported_seq":8190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.881467+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:04.881467+0000","last_peered":"2024-04-22T08:30:04.881467+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:04.881467+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8069,"log_dups_size":0,"ondisk_log_size":8069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33034240,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":19584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4310}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'8162","reported_seq":8190,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:30:04.875669+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:04.875669+0000","last_peered":"2024-04-22T08:30:04.875669+0000","last_clean":"2024-04-22T08:30:04.875669+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:04.875669+0000","last_undegraded":"2024-04-22T08:30:04.875669+0000","last_fullsized":"2024-04-22T08:30:04.875669+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8162,"log_dups_size":0,"ondisk_log_size":8162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33431552,"num_objects":8162,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'8096","reported_seq":8232,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:30:04.339437+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:04.339437+0000","last_peered":"2024-04-22T08:30:04.339437+0000","last_clean":"2024-04-22T08:30:04.339437+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:04.339437+0000","last_undegraded":"2024-04-22T08:30:04.339437+0000","last_fullsized":"2024-04-22T08:30:04.339437+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33132544,"num_objects":8089,"num_object_clones":0,"num_object_copies":16178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":20752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'8202","reported_seq":8362,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.346751+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:04.346751+0000","last_peered":"2024-04-22T08:30:04.346751+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:04.346751+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33574912,"num_objects":8197,"num_object_clones":0,"num_object_copies":16394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":20284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4318},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'8038","reported_seq":15364,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:04.410544+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:30:04.410544+0000","last_peered":"2024-04-22T08:30:04.410544+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:30:04.410544+0000","last_undegraded":"2024-04-22T08:30:04.410544+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":8032,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":199,"num_objects_unfound":0,"num_objects_dirty":8032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3584,"num_bytes_recovered":14671872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7833},{"shards":"0,2","objects":199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'8056","reported_seq":8198,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:30:04.397738+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:04.397738+0000","last_peered":"2024-04-22T08:30:04.397738+0000","last_clean":"2024-04-22T08:30:04.397738+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:04.397738+0000","last_undegraded":"2024-04-22T08:30:04.397738+0000","last_fullsized":"2024-04-22T08:30:04.397738+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32989184,"num_objects":8054,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4704,"num_write_kb":18816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"35'8249","reported_seq":8370,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.858058+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:04.858058+0000","last_peered":"2024-04-22T08:30:04.858058+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:04.858058+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8249,"log_dups_size":0,"ondisk_log_size":8249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33522898,"num_objects":8186,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8189,"num_write_kb":32740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4298}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.183376+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:27:56.183376+0000","last_peered":"2024-04-22T08:27:56.183376+0000","last_clean":"2024-04-22T08:27:56.183376+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:27:56.183376+0000","last_undegraded":"2024-04-22T08:27:56.183376+0000","last_fullsized":"2024-04-22T08:27:56.183376+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'8154","reported_seq":8229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.282473+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:04.282473+0000","last_peered":"2024-04-22T08:30:04.282473+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:04.282473+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8154,"log_dups_size":0,"ondisk_log_size":8154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33367250,"num_objects":8148,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":32588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4377},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.250381+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:27:56.250381+0000","last_peered":"2024-04-22T08:27:56.250381+0000","last_clean":"2024-04-22T08:27:56.250381+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:27:56.250381+0000","last_undegraded":"2024-04-22T08:27:56.250381+0000","last_fullsized":"2024-04-22T08:27:56.250381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'8085","reported_seq":8229,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.848059+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:04.848059+0000","last_peered":"2024-04-22T08:30:04.848059+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:04.848059+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8085,"log_dups_size":0,"ondisk_log_size":8085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33095680,"num_objects":8080,"num_object_clones":0,"num_object_copies":16160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":18156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"35'8129","reported_seq":8182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.839832+0000","last_change":"2024-04-22T08:27:56.632687+0000","last_active":"2024-04-22T08:30:04.839832+0000","last_peered":"2024-04-22T08:30:04.839832+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:04.839832+0000","last_undegraded":"2024-04-22T08:27:56.180186+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8129,"log_dups_size":0,"ondisk_log_size":8129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273042,"num_objects":8125,"num_object_clones":0,"num_object_copies":16250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8127,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3803},{"shards":"2","objects":4322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'8150","reported_seq":8224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.403583+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:04.403583+0000","last_peered":"2024-04-22T08:30:04.403583+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:04.403583+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33374825,"num_objects":8149,"num_object_clones":0,"num_object_copies":16298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8149,"num_write_kb":32593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4428},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"35'8023","reported_seq":8195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.412684+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:04.412684+0000","last_peered":"2024-04-22T08:30:04.412684+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:04.412684+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8023,"log_dups_size":0,"ondisk_log_size":8023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32829440,"num_objects":8015,"num_object_clones":0,"num_object_copies":16030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":17016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4178},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.091640+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:27:56.091640+0000","last_peered":"2024-04-22T08:27:56.091640+0000","last_clean":"2024-04-22T08:27:56.091640+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:27:56.091640+0000","last_undegraded":"2024-04-22T08:27:56.091640+0000","last_fullsized":"2024-04-22T08:27:56.091640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'8163","reported_seq":8237,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.381137+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:04.381137+0000","last_peered":"2024-04-22T08:30:04.381137+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:04.381137+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420498,"num_objects":8161,"num_object_clones":0,"num_object_copies":16322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8161,"num_write_kb":32638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4456},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.104008+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.104008+0000","last_peered":"2024-04-22T08:27:56.104008+0000","last_clean":"2024-04-22T08:27:56.104008+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.104008+0000","last_undegraded":"2024-04-22T08:27:56.104008+0000","last_fullsized":"2024-04-22T08:27:56.104008+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"35'8201","reported_seq":8343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:04.402137+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:04.402137+0000","last_peered":"2024-04-22T08:30:04.402137+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:04.402137+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33544612,"num_objects":8193,"num_object_clones":0,"num_object_copies":16386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":32763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4312},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":532807509,"num_objects":130091,"num_object_clones":0,"num_object_copies":260182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41804,"num_objects_misplaced":199,"num_objects_unfound":0,"num_objects_dirty":130091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103693,"num_write_kb":414685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3604,"num_bytes_recovered":14688873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077866496,"data_stored":1077786479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130218,"ondisk_log_size":130218,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117033,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579500,"kb_used_data":381348,"kb_used_omap":4,"kb_used_meta":198139,"kb_avail":93792340,"statfs":{"total":96636764160,"available":96043356160,"internally_reserved":0,"allocated":390500352,"data_stored":390299337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202895055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35799999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.26200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920795,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435432,"kb_used_data":219232,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93936408,"statfs":{"total":96636764160,"available":96190881792,"internally_reserved":0,"allocated":224493568,"data_stored":224222973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.0149999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.29499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738480,"kb_used_data":450152,"kb_used_omap":4,"kb_used_meta":288315,"kb_avail":93633360,"statfs":{"total":96636764160,"available":95880560640,"internally_reserved":0,"allocated":460955648,"data_stored":460761095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":295235283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.73799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":462036992,"data_stored":462033513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225193984,"data_stored":225120925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390635520,"data_stored":390632041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:07.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 175 20 131588 131568 2.93576 1.85156 0.0706685 0.0266035 2024-04-22T08:30:08.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 176 20 132072 132052 2.92983 1.89062 0.0144711 0.0266542 2024-04-22T08:30:09.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 177 20 132478 132458 2.92223 1.58594 0.00622618 0.0266698 2024-04-22T08:30:10.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:10.873 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 178 20 132786 132766 2.91258 1.20312 0.0635078 0.0268154 2024-04-22T08:30:11.458 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:11.459 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:11.530 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-04-22T08:30:10.205625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541699925,"num_objects":132262,"num_object_clones":0,"num_object_copies":264524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3959,"num_objects_unfound":0,"num_objects_dirty":132262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105864,"num_write_kb":423369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":14865001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132389,"ondisk_log_size":132389,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1821188,"kb_used_data":1070120,"kb_used_omap":12,"kb_used_meta":751027,"kb_avail":281294332,"statfs":{"total":289910292480,"available":288045395968,"internally_reserved":0,"allocated":1095802880,"data_stored":1095131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":769051758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,8,4,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18513920,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3805,"num_objects_misplaced":3568,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":962560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002109"},"pg_stats":[{"pgid":"2.f","version":"36'8356","reported_seq":8418,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.462376+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:09.462376+0000","last_peered":"2024-04-22T08:30:09.462376+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:09.462376+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34226176,"num_objects":8356,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":33424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4562},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"36'8350","reported_seq":8464,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.675746+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:09.675746+0000","last_peered":"2024-04-22T08:30:09.675746+0000","last_clean":"2024-04-22T08:30:09.675746+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:09.675746+0000","last_undegraded":"2024-04-22T08:30:09.675746+0000","last_fullsized":"2024-04-22T08:30:09.675746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34177024,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":21624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"36'8206","reported_seq":8328,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.833846+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:09.833846+0000","last_peered":"2024-04-22T08:30:09.833846+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:09.833846+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33595392,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":20132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"36'8311","reported_seq":8340,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.864692+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:09.864692+0000","last_peered":"2024-04-22T08:30:09.864692+0000","last_clean":"2024-04-22T08:30:09.864692+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:09.864692+0000","last_undegraded":"2024-04-22T08:30:09.864692+0000","last_fullsized":"2024-04-22T08:30:09.864692+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8311,"log_dups_size":0,"ondisk_log_size":8311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34041856,"num_objects":8311,"num_object_clones":0,"num_object_copies":16622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"36'8261","reported_seq":8398,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.359728+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:09.359728+0000","last_peered":"2024-04-22T08:30:09.359728+0000","last_clean":"2024-04-22T08:30:09.359728+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:09.359728+0000","last_undegraded":"2024-04-22T08:30:09.359728+0000","last_fullsized":"2024-04-22T08:30:09.359728+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":21412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"36'8329","reported_seq":8490,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.421318+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:09.421318+0000","last_peered":"2024-04-22T08:30:09.421318+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:09.421318+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34095104,"num_objects":8324,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4445},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'8038","reported_seq":15364,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:04.410544+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:30:04.410544+0000","last_peered":"2024-04-22T08:30:04.410544+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:30:04.410544+0000","last_undegraded":"2024-04-22T08:30:04.410544+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":8032,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":199,"num_objects_unfound":0,"num_objects_dirty":8032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3584,"num_bytes_recovered":14671872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7833},{"shards":"0,2","objects":199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"36'8198","reported_seq":8341,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.441722+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:09.441722+0000","last_peered":"2024-04-22T08:30:09.441722+0000","last_clean":"2024-04-22T08:30:09.441722+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:09.441722+0000","last_undegraded":"2024-04-22T08:30:09.441722+0000","last_fullsized":"2024-04-22T08:30:09.441722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8198,"log_dups_size":0,"ondisk_log_size":8198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33570816,"num_objects":8196,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4846,"num_write_kb":19384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'8402","reported_seq":8524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.821326+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:09.821326+0000","last_peered":"2024-04-22T08:30:09.821326+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:09.821326+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34149586,"num_objects":8339,"num_object_clones":0,"num_object_copies":16678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":33352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.693201+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:29:09.693201+0000","last_peered":"2024-04-22T08:29:09.693201+0000","last_clean":"2024-04-22T08:29:09.693201+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:29:09.693201+0000","last_undegraded":"2024-04-22T08:29:09.693201+0000","last_fullsized":"2024-04-22T08:29:09.693201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"36'8304","reported_seq":8380,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.435846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:09.435846+0000","last_peered":"2024-04-22T08:30:09.435846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:09.435846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33981650,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":33188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4527},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.851476+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:28:53.851476+0000","last_peered":"2024-04-22T08:28:53.851476+0000","last_clean":"2024-04-22T08:28:53.851476+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:28:53.851476+0000","last_undegraded":"2024-04-22T08:28:53.851476+0000","last_fullsized":"2024-04-22T08:28:53.851476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"36'8222","reported_seq":8367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.771548+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:09.771548+0000","last_peered":"2024-04-22T08:30:09.771548+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:09.771548+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33656832,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"36'8273","reported_seq":8414,"reported_epoch":36,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:09.917974+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:09.917974+0000","last_peered":"2024-04-22T08:30:09.917974+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:09.917974+0000","last_undegraded":"2024-04-22T08:30:09.917974+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33862866,"num_objects":8269,"num_object_clones":0,"num_object_copies":16538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3760,"num_objects_unfound":0,"num_objects_dirty":8269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":176128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3760},{"shards":"2","objects":4509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"36'8310","reported_seq":8388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.395483+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:09.395483+0000","last_peered":"2024-04-22T08:30:09.395483+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:09.395483+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34030185,"num_objects":8309,"num_object_clones":0,"num_object_copies":16618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8309,"num_write_kb":33233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4588},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"36'8175","reported_seq":8349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.354970+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:09.354970+0000","last_peered":"2024-04-22T08:30:09.354970+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:09.354970+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33452032,"num_objects":8167,"num_object_clones":0,"num_object_copies":16334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4330},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.705157+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:29:09.705157+0000","last_peered":"2024-04-22T08:29:09.705157+0000","last_clean":"2024-04-22T08:29:09.705157+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:29:09.705157+0000","last_undegraded":"2024-04-22T08:29:09.705157+0000","last_fullsized":"2024-04-22T08:29:09.705157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"36'8308","reported_seq":8383,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.269713+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:09.269713+0000","last_peered":"2024-04-22T08:30:09.269713+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:09.269713+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34014418,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":33218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.104008+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.104008+0000","last_peered":"2024-04-22T08:27:56.104008+0000","last_clean":"2024-04-22T08:27:56.104008+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.104008+0000","last_undegraded":"2024-04-22T08:27:56.104008+0000","last_fullsized":"2024-04-22T08:27:56.104008+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"36'8346","reported_seq":8489,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.393323+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:09.393323+0000","last_peered":"2024-04-22T08:30:09.393323+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:09.393323+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34138532,"num_objects":8338,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":33343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4457},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":541699925,"num_objects":132262,"num_object_clones":0,"num_object_copies":264524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3959,"num_objects_unfound":0,"num_objects_dirty":132262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105864,"num_write_kb":423369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":14865001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097846784,"data_stored":1097766767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132389,"ondisk_log_size":132389,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624488,"kb_used_data":386140,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93747352,"statfs":{"total":96636764160,"available":95997288448,"internally_reserved":0,"allocated":395407360,"data_stored":395201158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244051663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.252},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":1.3560000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920796,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447140,"kb_used_data":226844,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93924700,"statfs":{"total":96636764160,"available":96178892800,"internally_reserved":0,"allocated":232288256,"data_stored":232017661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":225570508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.221},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":7.3239999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.25}]}]},{"osd":0,"up_from":21,"seq":90194313262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749560,"kb_used_data":457136,"kb_used_omap":4,"kb_used_meta":292411,"kb_avail":93622280,"statfs":{"total":96636764160,"available":95869214720,"internally_reserved":0,"allocated":468107264,"data_stored":467912711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299429587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":8.1880000000000006},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":6.9580000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":469204992,"data_stored":469201513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233328640,"data_stored":233255581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395313152,"data_stored":395309673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:11.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:11.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 179 20 133247 133227 2.90636 1.80078 0.060908 0.0268738 2024-04-22T08:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:12.428 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:12.503 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-04-22T08:30:12.206355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541699925,"num_objects":132262,"num_object_clones":0,"num_object_copies":264524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3959,"num_objects_unfound":0,"num_objects_dirty":132262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105864,"num_write_kb":423369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":14865001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132389,"ondisk_log_size":132389,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1821188,"kb_used_data":1070120,"kb_used_omap":12,"kb_used_meta":751027,"kb_avail":281294332,"statfs":{"total":289910292480,"available":288045395968,"internally_reserved":0,"allocated":1095802880,"data_stored":1095131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":769051758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,8,4,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18513920,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3805,"num_objects_misplaced":3568,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":962560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002279"},"pg_stats":[{"pgid":"2.f","version":"36'8356","reported_seq":8418,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.462376+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:09.462376+0000","last_peered":"2024-04-22T08:30:09.462376+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:09.462376+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34226176,"num_objects":8356,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":33424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4562},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"36'8350","reported_seq":8464,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.675746+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:09.675746+0000","last_peered":"2024-04-22T08:30:09.675746+0000","last_clean":"2024-04-22T08:30:09.675746+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:09.675746+0000","last_undegraded":"2024-04-22T08:30:09.675746+0000","last_fullsized":"2024-04-22T08:30:09.675746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34177024,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":21624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"36'8206","reported_seq":8328,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.833846+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:09.833846+0000","last_peered":"2024-04-22T08:30:09.833846+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:09.833846+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33595392,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":20132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"36'8311","reported_seq":8340,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.864692+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:09.864692+0000","last_peered":"2024-04-22T08:30:09.864692+0000","last_clean":"2024-04-22T08:30:09.864692+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:09.864692+0000","last_undegraded":"2024-04-22T08:30:09.864692+0000","last_fullsized":"2024-04-22T08:30:09.864692+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8311,"log_dups_size":0,"ondisk_log_size":8311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34041856,"num_objects":8311,"num_object_clones":0,"num_object_copies":16622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"36'8261","reported_seq":8398,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.359728+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:09.359728+0000","last_peered":"2024-04-22T08:30:09.359728+0000","last_clean":"2024-04-22T08:30:09.359728+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:09.359728+0000","last_undegraded":"2024-04-22T08:30:09.359728+0000","last_fullsized":"2024-04-22T08:30:09.359728+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":21412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"36'8329","reported_seq":8490,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.421318+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:09.421318+0000","last_peered":"2024-04-22T08:30:09.421318+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:09.421318+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34095104,"num_objects":8324,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4445},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'8038","reported_seq":15364,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:04.410544+0000","last_change":"2024-04-22T08:27:56.611075+0000","last_active":"2024-04-22T08:30:04.410544+0000","last_peered":"2024-04-22T08:30:04.410544+0000","last_clean":"2024-04-22T08:27:54.063128+0000","last_became_active":"2024-04-22T08:27:56.556354+0000","last_became_peered":"2024-04-22T08:27:56.556354+0000","last_unstale":"2024-04-22T08:30:04.410544+0000","last_undegraded":"2024-04-22T08:30:04.410544+0000","last_fullsized":"2024-04-22T08:27:56.083529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":8032,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":199,"num_objects_unfound":0,"num_objects_dirty":8032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3584,"num_bytes_recovered":14671872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7833},{"shards":"0,2","objects":199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"36'8198","reported_seq":8341,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.441722+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:09.441722+0000","last_peered":"2024-04-22T08:30:09.441722+0000","last_clean":"2024-04-22T08:30:09.441722+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:09.441722+0000","last_undegraded":"2024-04-22T08:30:09.441722+0000","last_fullsized":"2024-04-22T08:30:09.441722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8198,"log_dups_size":0,"ondisk_log_size":8198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33570816,"num_objects":8196,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4846,"num_write_kb":19384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.107832+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:27:56.107832+0000","last_peered":"2024-04-22T08:27:56.107832+0000","last_clean":"2024-04-22T08:27:56.107832+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:27:56.107832+0000","last_undegraded":"2024-04-22T08:27:56.107832+0000","last_fullsized":"2024-04-22T08:27:56.107832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'8402","reported_seq":8524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.821326+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:09.821326+0000","last_peered":"2024-04-22T08:30:09.821326+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:09.821326+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34149586,"num_objects":8339,"num_object_clones":0,"num_object_copies":16678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":33352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.693201+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:29:09.693201+0000","last_peered":"2024-04-22T08:29:09.693201+0000","last_clean":"2024-04-22T08:29:09.693201+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:29:09.693201+0000","last_undegraded":"2024-04-22T08:29:09.693201+0000","last_fullsized":"2024-04-22T08:29:09.693201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"36'8304","reported_seq":8380,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.435846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:09.435846+0000","last_peered":"2024-04-22T08:30:09.435846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:09.435846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33981650,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":33188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4527},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.851476+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:28:53.851476+0000","last_peered":"2024-04-22T08:28:53.851476+0000","last_clean":"2024-04-22T08:28:53.851476+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:28:53.851476+0000","last_undegraded":"2024-04-22T08:28:53.851476+0000","last_fullsized":"2024-04-22T08:28:53.851476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"36'8222","reported_seq":8367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.771548+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:09.771548+0000","last_peered":"2024-04-22T08:30:09.771548+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:09.771548+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33656832,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.646648+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:27:56.646648+0000","last_peered":"2024-04-22T08:27:56.646648+0000","last_clean":"2024-04-22T08:27:56.646648+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:27:56.646648+0000","last_undegraded":"2024-04-22T08:27:56.646648+0000","last_fullsized":"2024-04-22T08:27:56.646648+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"36'8273","reported_seq":8414,"reported_epoch":36,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:09.917974+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:09.917974+0000","last_peered":"2024-04-22T08:30:09.917974+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:09.917974+0000","last_undegraded":"2024-04-22T08:30:09.917974+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33862866,"num_objects":8269,"num_object_clones":0,"num_object_copies":16538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3760,"num_objects_unfound":0,"num_objects_dirty":8269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":176128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3760},{"shards":"2","objects":4509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"36'8310","reported_seq":8388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.395483+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:09.395483+0000","last_peered":"2024-04-22T08:30:09.395483+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:09.395483+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34030185,"num_objects":8309,"num_object_clones":0,"num_object_copies":16618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8309,"num_write_kb":33233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4588},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.651432+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:27:56.651432+0000","last_peered":"2024-04-22T08:27:56.651432+0000","last_clean":"2024-04-22T08:27:56.651432+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:27:56.651432+0000","last_undegraded":"2024-04-22T08:27:56.651432+0000","last_fullsized":"2024-04-22T08:27:56.651432+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"36'8175","reported_seq":8349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.354970+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:09.354970+0000","last_peered":"2024-04-22T08:30:09.354970+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:09.354970+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33452032,"num_objects":8167,"num_object_clones":0,"num_object_copies":16334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4330},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.705157+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:29:09.705157+0000","last_peered":"2024-04-22T08:29:09.705157+0000","last_clean":"2024-04-22T08:29:09.705157+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:29:09.705157+0000","last_undegraded":"2024-04-22T08:29:09.705157+0000","last_fullsized":"2024-04-22T08:29:09.705157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"36'8308","reported_seq":8383,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.269713+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:09.269713+0000","last_peered":"2024-04-22T08:30:09.269713+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:09.269713+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34014418,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":33218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.666446+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:27:56.666446+0000","last_peered":"2024-04-22T08:27:56.666446+0000","last_clean":"2024-04-22T08:27:56.666446+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:27:56.666446+0000","last_undegraded":"2024-04-22T08:27:56.666446+0000","last_fullsized":"2024-04-22T08:27:56.666446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:27:56.104008+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:27:56.104008+0000","last_peered":"2024-04-22T08:27:56.104008+0000","last_clean":"2024-04-22T08:27:56.104008+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:27:56.104008+0000","last_undegraded":"2024-04-22T08:27:56.104008+0000","last_fullsized":"2024-04-22T08:27:56.104008+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"36'8346","reported_seq":8489,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.393323+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:09.393323+0000","last_peered":"2024-04-22T08:30:09.393323+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:09.393323+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34138532,"num_objects":8338,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":33343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4457},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":541699925,"num_objects":132262,"num_object_clones":0,"num_object_copies":264524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3959,"num_objects_unfound":0,"num_objects_dirty":132262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105864,"num_write_kb":423369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3647,"num_bytes_recovered":14865001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097846784,"data_stored":1097766767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132389,"ondisk_log_size":132389,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624488,"kb_used_data":386140,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93747352,"statfs":{"total":96636764160,"available":95997288448,"internally_reserved":0,"allocated":395407360,"data_stored":395201158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244051663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.252},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":1.3560000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920796,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447140,"kb_used_data":226844,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93924700,"statfs":{"total":96636764160,"available":96178892800,"internally_reserved":0,"allocated":232288256,"data_stored":232017661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":225570508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.221},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":7.3239999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.25}]}]},{"osd":0,"up_from":21,"seq":90194313262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749560,"kb_used_data":457136,"kb_used_omap":4,"kb_used_meta":292411,"kb_avail":93622280,"statfs":{"total":96636764160,"available":95869214720,"internally_reserved":0,"allocated":468107264,"data_stored":467912711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299429587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":8.1880000000000006},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":6.9580000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":469204992,"data_stored":469201513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233328640,"data_stored":233255581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395313152,"data_stored":395309673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "36'8356", 'reported_seq': 8418, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.462376+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:09.462376+0000', 'last_peered': '2024-04-22T08:30:09.462376+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:09.462376+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8356, 'log_dups_size': 0, 'ondisk_log_size': 8356, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34226176, 'num_objects': 8356, 'num_object_clones': 0, 'num_object_copies': 16712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8356, 'num_write_kb': 33424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4562}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "36'8206", 'reported_seq': 8328, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.833846+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:09.833846+0000', 'last_peered': '2024-04-22T08:30:09.833846+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:09.833846+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 8206, 'log_dups_size': 0, 'ondisk_log_size': 8206, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33595392, 'num_objects': 8202, 'num_object_clones': 0, 'num_object_copies': 16404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5033, 'num_write_kb': 20132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 4447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:12.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "36'8329", 'reported_seq': 8490, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.421318+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:09.421318+0000', 'last_peered': '2024-04-22T08:30:09.421318+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:09.421318+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 8329, 'log_dups_size': 0, 'ondisk_log_size': 8329, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34095104, 'num_objects': 8324, 'num_object_clones': 0, 'num_object_copies': 16648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5198, 'num_write_kb': 20792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4445}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "35'8038", 'reported_seq': 15364, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:04.410544+0000', 'last_change': '2024-04-22T08:27:56.611075+0000', 'last_active': '2024-04-22T08:30:04.410544+0000', 'last_peered': '2024-04-22T08:30:04.410544+0000', 'last_clean': '2024-04-22T08:27:54.063128+0000', 'last_became_active': '2024-04-22T08:27:56.556354+0000', 'last_became_peered': '2024-04-22T08:27:56.556354+0000', 'last_unstale': '2024-04-22T08:30:04.410544+0000', 'last_undegraded': '2024-04-22T08:30:04.410544+0000', 'last_fullsized': '2024-04-22T08:27:56.083529+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3707", 'last_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.930922+0000', 'objects_scrubbed': 3626, 'log_size': 8038, 'log_dups_size': 0, 'ondisk_log_size': 8038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:22:07.652675+0000', 'scrub_duration': 2.305596695, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32899072, 'num_objects': 8032, 'num_object_clones': 0, 'num_object_copies': 16064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 199, 'num_objects_unfound': 0, 'num_objects_dirty': 8032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4406, 'num_write_kb': 17624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3584, 'num_bytes_recovered': 14671872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7833}, {'shards': '0,2', 'objects': 199}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "36'8402", 'reported_seq': 8524, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.821326+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:09.821326+0000', 'last_peered': '2024-04-22T08:30:09.821326+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:09.821326+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 8402, 'log_dups_size': 0, 'ondisk_log_size': 8402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34149586, 'num_objects': 8339, 'num_object_clones': 0, 'num_object_copies': 16678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8342, 'num_write_kb': 33352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 4451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "36'8304", 'reported_seq': 8380, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.435846+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:09.435846+0000', 'last_peered': '2024-04-22T08:30:09.435846+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:09.435846+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8304, 'log_dups_size': 0, 'ondisk_log_size': 8304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33981650, 'num_objects': 8298, 'num_object_clones': 0, 'num_object_copies': 16596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8302, 'num_write_kb': 33188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4527}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "36'8222", 'reported_seq': 8367, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.771548+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:09.771548+0000', 'last_peered': '2024-04-22T08:30:09.771548+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:09.771548+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 8222, 'log_dups_size': 0, 'ondisk_log_size': 8222, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33656832, 'num_objects': 8217, 'num_object_clones': 0, 'num_object_copies': 16434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4676, 'num_write_kb': 18704, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 4467}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:12.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "36'8273", 'reported_seq': 8414, 'reported_epoch': 36, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:09.917974+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:09.917974+0000', 'last_peered': '2024-04-22T08:30:09.917974+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:09.917974+0000', 'last_undegraded': '2024-04-22T08:30:09.917974+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8273, 'log_dups_size': 0, 'ondisk_log_size': 8273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33862866, 'num_objects': 8269, 'num_object_clones': 0, 'num_object_copies': 16538, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3760, 'num_objects_unfound': 0, 'num_objects_dirty': 8269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8271, 'num_write_kb': 33071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 176128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3760}, {'shards': '2', 'objects': 4509}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "36'8310", 'reported_seq': 8388, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.395483+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:09.395483+0000', 'last_peered': '2024-04-22T08:30:09.395483+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:09.395483+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8310, 'log_dups_size': 0, 'ondisk_log_size': 8310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34030185, 'num_objects': 8309, 'num_object_clones': 0, 'num_object_copies': 16618, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8309, 'num_write_kb': 33233, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4588}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "36'8175", 'reported_seq': 8349, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.354970+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:09.354970+0000', 'last_peered': '2024-04-22T08:30:09.354970+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:09.354970+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8175, 'log_dups_size': 0, 'ondisk_log_size': 8175, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33452032, 'num_objects': 8167, 'num_object_clones': 0, 'num_object_copies': 16334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4406, 'num_write_kb': 17624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4330}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "36'8308", 'reported_seq': 8383, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.269713+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:09.269713+0000', 'last_peered': '2024-04-22T08:30:09.269713+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:09.269713+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8308, 'log_dups_size': 0, 'ondisk_log_size': 8308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34014418, 'num_objects': 8306, 'num_object_clones': 0, 'num_object_copies': 16612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8306, 'num_write_kb': 33218, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4601}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:12.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "36'8346", 'reported_seq': 8489, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:09.393323+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:09.393323+0000', 'last_peered': '2024-04-22T08:30:09.393323+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:09.393323+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 8346, 'log_dups_size': 0, 'ondisk_log_size': 8346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34138532, 'num_objects': 8338, 'num_object_clones': 0, 'num_object_copies': 16676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8343, 'num_write_kb': 33343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4457}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:12.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:12.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:30:12.876126+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0269315 2024-04-22T08:30:12.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:30:12.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 180 19 133704 133685 2.90015 1.78906 0.0274675 0.0269315 2024-04-22T08:30:13.619 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:13.619 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/264524 objects degraded (14.365%), 10 pgs degraded, 12 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":248,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":24,"num_pools":2,"num_objects":132262,"data_bytes":541699925,"bytes_used":1864896512,"bytes_avail":288045395968,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":264524,"degraded_ratio":0.14365048161981522,"misplaced_objects":3959,"misplaced_total":264524,"misplaced_ratio":0.01496650587470324,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":120285,"recovering_keys_per_sec":0,"num_objects_recovered":235,"num_bytes_recovered":962560,"num_keys_recovered":0,"write_bytes_sec":2313580,"read_op_per_sec":0,"write_op_per_sec":564},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-22T08:30:02.203547+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:30:13.655 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:13.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:13.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 181 20 134160 134140 2.89395 1.77734 0.041588 0.0269869 2024-04-22T08:30:14.535 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:14.536 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:14.603 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-04-22T08:30:14.206815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542699349,"num_objects":132506,"num_object_clones":0,"num_object_copies":265012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3760,"num_objects_unfound":0,"num_objects_dirty":132506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106108,"num_write_kb":424345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3846,"num_bytes_recovered":15680105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132633,"ondisk_log_size":132633,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1820128,"kb_used_data":1077252,"kb_used_omap":12,"kb_used_meta":742835,"kb_avail":281295392,"statfs":{"total":289910292480,"available":288046481408,"internally_reserved":0,"allocated":1103106048,"data_stored":1102424303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":760663150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,5,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9891840,"num_objects":2415,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3805,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":9660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002236"},"pg_stats":[{"pgid":"2.f","version":"36'8356","reported_seq":8418,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.462376+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:09.462376+0000","last_peered":"2024-04-22T08:30:09.462376+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:09.462376+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34226176,"num_objects":8356,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":33424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4562},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"36'8350","reported_seq":8464,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.675746+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:09.675746+0000","last_peered":"2024-04-22T08:30:09.675746+0000","last_clean":"2024-04-22T08:30:09.675746+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:09.675746+0000","last_undegraded":"2024-04-22T08:30:09.675746+0000","last_fullsized":"2024-04-22T08:30:09.675746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34177024,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":21624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"36'8206","reported_seq":8328,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.833846+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:09.833846+0000","last_peered":"2024-04-22T08:30:09.833846+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:09.833846+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33595392,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":20132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"36'8311","reported_seq":8340,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.864692+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:09.864692+0000","last_peered":"2024-04-22T08:30:09.864692+0000","last_clean":"2024-04-22T08:30:09.864692+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:09.864692+0000","last_undegraded":"2024-04-22T08:30:09.864692+0000","last_fullsized":"2024-04-22T08:30:09.864692+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8311,"log_dups_size":0,"ondisk_log_size":8311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34041856,"num_objects":8311,"num_object_clones":0,"num_object_copies":16622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"36'8261","reported_seq":8398,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.359728+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:09.359728+0000","last_peered":"2024-04-22T08:30:09.359728+0000","last_clean":"2024-04-22T08:30:09.359728+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:09.359728+0000","last_undegraded":"2024-04-22T08:30:09.359728+0000","last_fullsized":"2024-04-22T08:30:09.359728+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":21412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"36'8329","reported_seq":8490,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.421318+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:09.421318+0000","last_peered":"2024-04-22T08:30:09.421318+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:09.421318+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34095104,"num_objects":8324,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":20792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4445},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8282","reported_seq":16018,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:13.036634+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:13.036634+0000","last_peered":"2024-04-22T08:30:13.036634+0000","last_clean":"2024-04-22T08:30:13.036634+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:13.036634+0000","last_undegraded":"2024-04-22T08:30:13.036634+0000","last_fullsized":"2024-04-22T08:30:13.036634+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33898496,"num_objects":8276,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":18600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"36'8198","reported_seq":8341,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.441722+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:09.441722+0000","last_peered":"2024-04-22T08:30:09.441722+0000","last_clean":"2024-04-22T08:30:09.441722+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:09.441722+0000","last_undegraded":"2024-04-22T08:30:09.441722+0000","last_fullsized":"2024-04-22T08:30:09.441722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8198,"log_dups_size":0,"ondisk_log_size":8198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33570816,"num_objects":8196,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4846,"num_write_kb":19384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'8402","reported_seq":8524,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.821326+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:09.821326+0000","last_peered":"2024-04-22T08:30:09.821326+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:09.821326+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34149586,"num_objects":8339,"num_object_clones":0,"num_object_copies":16678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":33352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.693201+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:29:09.693201+0000","last_peered":"2024-04-22T08:29:09.693201+0000","last_clean":"2024-04-22T08:29:09.693201+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:29:09.693201+0000","last_undegraded":"2024-04-22T08:29:09.693201+0000","last_fullsized":"2024-04-22T08:29:09.693201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"36'8304","reported_seq":8380,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.435846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:09.435846+0000","last_peered":"2024-04-22T08:30:09.435846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:09.435846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33981650,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":33188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4527},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:28:53.851476+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:28:53.851476+0000","last_peered":"2024-04-22T08:28:53.851476+0000","last_clean":"2024-04-22T08:28:53.851476+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:28:53.851476+0000","last_undegraded":"2024-04-22T08:28:53.851476+0000","last_fullsized":"2024-04-22T08:28:53.851476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"36'8222","reported_seq":8367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.771548+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:09.771548+0000","last_peered":"2024-04-22T08:30:09.771548+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:09.771548+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33656832,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"36'8273","reported_seq":8414,"reported_epoch":36,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:09.917974+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:09.917974+0000","last_peered":"2024-04-22T08:30:09.917974+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:09.917974+0000","last_undegraded":"2024-04-22T08:30:09.917974+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33862866,"num_objects":8269,"num_object_clones":0,"num_object_copies":16538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3760,"num_objects_unfound":0,"num_objects_dirty":8269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":176128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3760},{"shards":"2","objects":4509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"36'8310","reported_seq":8388,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.395483+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:09.395483+0000","last_peered":"2024-04-22T08:30:09.395483+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:09.395483+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8310,"log_dups_size":0,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34030185,"num_objects":8309,"num_object_clones":0,"num_object_copies":16618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8309,"num_write_kb":33233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4588},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"36'8175","reported_seq":8349,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.354970+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:09.354970+0000","last_peered":"2024-04-22T08:30:09.354970+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:09.354970+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33452032,"num_objects":8167,"num_object_clones":0,"num_object_copies":16334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":17624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4330},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T08:29:09.705157+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:29:09.705157+0000","last_peered":"2024-04-22T08:29:09.705157+0000","last_clean":"2024-04-22T08:29:09.705157+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:29:09.705157+0000","last_undegraded":"2024-04-22T08:29:09.705157+0000","last_fullsized":"2024-04-22T08:29:09.705157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"36'8308","reported_seq":8383,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.269713+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:09.269713+0000","last_peered":"2024-04-22T08:30:09.269713+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:09.269713+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34014418,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":33218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"36'8346","reported_seq":8489,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:09.393323+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:09.393323+0000","last_peered":"2024-04-22T08:30:09.393323+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:09.393323+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34138532,"num_objects":8338,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":33343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4457},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":542699349,"num_objects":132506,"num_object_clones":0,"num_object_copies":265012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3760,"num_objects_unfound":0,"num_objects_dirty":132506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106108,"num_write_kb":424345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3846,"num_bytes_recovered":15680105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105121280,"data_stored":1105041263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132633,"ondisk_log_size":132633,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624488,"kb_used_data":386140,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93747352,"statfs":{"total":96636764160,"available":95997288448,"internally_reserved":0,"allocated":395407360,"data_stored":395201158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244051663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.78200000000000003},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.0839999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.159,"5min":0.65900000000000003,"15min":0.57599999999999996},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.252},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":1.3560000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920797,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446080,"kb_used_data":233976,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93925760,"statfs":{"total":96636764160,"available":96179978240,"internally_reserved":0,"allocated":239591424,"data_stored":239310434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":217181900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.7569999999999997},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.26100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749560,"kb_used_data":457136,"kb_used_omap":4,"kb_used_meta":292411,"kb_avail":93622280,"statfs":{"total":96636764160,"available":95869214720,"internally_reserved":0,"allocated":468107264,"data_stored":467912711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":299429587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":8.1880000000000006},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.22,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":6.9580000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.81999999999999995,"15min":0.79200000000000004},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":469204992,"data_stored":469201513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240603136,"data_stored":240530077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395313152,"data_stored":395309673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:14.874 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 182 20 134626 134606 2.88805 1.82031 0.0647576 0.0270419 2024-04-22T08:30:15.877 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 183 20 135112 135092 2.88264 1.89844 0.0264752 0.0270946 2024-04-22T08:30:16.879 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 184 20 135612 135592 2.87752 1.95312 0.0902566 0.0271425 2024-04-22T08:30:17.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:17.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 185 20 136073 136053 2.87167 1.80078 0.0233529 0.0271955 2024-04-22T08:30:18.361 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:18.361 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:18.439 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-22T08:30:18.207818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":551509845,"num_objects":134657,"num_object_clones":0,"num_object_copies":269314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":134657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108259,"num_write_kb":432949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3943,"num_bytes_recovered":16077417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134784,"ondisk_log_size":134784,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1857748,"kb_used_data":1095164,"kb_used_omap":12,"kb_used_meta":762547,"kb_avail":281257772,"statfs":{"total":289910292480,"available":288007958528,"internally_reserved":0,"allocated":1121447936,"data_stored":1120750588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":780848238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18702336,"num_objects":4566,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3805,"num_objects_misplaced":3464,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":18264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":1388544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.071409"},"pg_stats":[{"pgid":"2.f","version":"37'8483","reported_seq":8546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.455755+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:14.455755+0000","last_peered":"2024-04-22T08:30:14.455755+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:14.455755+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8483,"log_dups_size":0,"ondisk_log_size":8483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34746368,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":33932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4689},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8485","reported_seq":8600,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.935859+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:14.935859+0000","last_peered":"2024-04-22T08:30:14.935859+0000","last_clean":"2024-04-22T08:30:14.935859+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:14.935859+0000","last_undegraded":"2024-04-22T08:30:14.935859+0000","last_fullsized":"2024-04-22T08:30:14.935859+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34729984,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8347","reported_seq":8470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.845574+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:14.845574+0000","last_peered":"2024-04-22T08:30:14.845574+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:14.845574+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34172928,"num_objects":8343,"num_object_clones":0,"num_object_copies":16686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":20696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8436","reported_seq":8466,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.859139+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:14.859139+0000","last_peered":"2024-04-22T08:30:14.859139+0000","last_clean":"2024-04-22T08:30:14.859139+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:14.859139+0000","last_undegraded":"2024-04-22T08:30:14.859139+0000","last_fullsized":"2024-04-22T08:30:14.859139+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34553856,"num_objects":8436,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8375","reported_seq":8513,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.475537+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:14.475537+0000","last_peered":"2024-04-22T08:30:14.475537+0000","last_clean":"2024-04-22T08:30:14.475537+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:14.475537+0000","last_undegraded":"2024-04-22T08:30:14.475537+0000","last_fullsized":"2024-04-22T08:30:14.475537+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275328,"num_objects":8368,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8464","reported_seq":8626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.344798+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:14.344798+0000","last_peered":"2024-04-22T08:30:14.344798+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:14.344798+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8464,"log_dups_size":0,"ondisk_log_size":8464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34648064,"num_objects":8459,"num_object_clones":0,"num_object_copies":16918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":21332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4580},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8467","reported_seq":16203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:18.058294+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:18.058294+0000","last_peered":"2024-04-22T08:30:18.058294+0000","last_clean":"2024-04-22T08:30:18.058294+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:18.058294+0000","last_undegraded":"2024-04-22T08:30:18.058294+0000","last_fullsized":"2024-04-22T08:30:18.058294+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34656256,"num_objects":8461,"num_object_clones":0,"num_object_copies":16922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8318","reported_seq":8461,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.435408+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:14.435408+0000","last_peered":"2024-04-22T08:30:14.435408+0000","last_clean":"2024-04-22T08:30:14.435408+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:14.435408+0000","last_undegraded":"2024-04-22T08:30:14.435408+0000","last_fullsized":"2024-04-22T08:30:14.435408+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34062336,"num_objects":8316,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":19864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8551","reported_seq":8674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.887855+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:14.887855+0000","last_peered":"2024-04-22T08:30:14.887855+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:14.887855+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34759890,"num_objects":8488,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8491,"num_write_kb":33948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8430","reported_seq":8507,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.323467+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:14.323467+0000","last_peered":"2024-04-22T08:30:14.323467+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:14.323467+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34497746,"num_objects":8424,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":33692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4653},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8370","reported_seq":8516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.703641+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:14.703641+0000","last_peered":"2024-04-22T08:30:14.703641+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:14.703641+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34263040,"num_objects":8365,"num_object_clones":0,"num_object_copies":16730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":19296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8411","reported_seq":8747,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:14.914197+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:14.914197+0000","last_peered":"2024-04-22T08:30:14.914197+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:14.914197+0000","last_undegraded":"2024-04-22T08:30:14.914197+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8411,"log_dups_size":0,"ondisk_log_size":8411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34428114,"num_objects":8407,"num_object_clones":0,"num_object_copies":16814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":8407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8409,"num_write_kb":33623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3663},{"shards":"2","objects":4744}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8438","reported_seq":8517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.453776+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:14.453776+0000","last_peered":"2024-04-22T08:30:14.453776+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:14.453776+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8438,"log_dups_size":0,"ondisk_log_size":8438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34554473,"num_objects":8437,"num_object_clones":0,"num_object_copies":16874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":33745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4716},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8304","reported_seq":8479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.428095+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:14.428095+0000","last_peered":"2024-04-22T08:30:14.428095+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:14.428095+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33980416,"num_objects":8296,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4535,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4459},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8434","reported_seq":8510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.415645+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:14.415645+0000","last_peered":"2024-04-22T08:30:14.415645+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:14.415645+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34530514,"num_objects":8432,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":33722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4727},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8471","reported_seq":8615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.408874+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:14.408874+0000","last_peered":"2024-04-22T08:30:14.408874+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:14.408874+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650532,"num_objects":8463,"num_object_clones":0,"num_object_copies":16926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":33843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4582},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":551509845,"num_objects":134657,"num_object_clones":0,"num_object_copies":269314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":134657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108259,"num_write_kb":432949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3943,"num_bytes_recovered":16077417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123532800,"data_stored":1123452783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134784,"ondisk_log_size":134784,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634424,"kb_used_data":390320,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93737416,"statfs":{"total":96636764160,"available":95987113984,"internally_reserved":0,"allocated":399687680,"data_stored":399476270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.373}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":4.2069999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920798,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457708,"kb_used_data":241508,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93914132,"statfs":{"total":96636764160,"available":96168071168,"internally_reserved":0,"allocated":247304192,"data_stored":247023202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313263,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765616,"kb_used_data":463336,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93606224,"statfs":{"total":96636764160,"available":95852773376,"internally_reserved":0,"allocated":474456064,"data_stored":474251116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":309522131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.875},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.84199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":475459584,"data_stored":475456105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248295424,"data_stored":248222365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399777792,"data_stored":399774313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:18.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:18.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 186 20 136514 136494 2.8655 1.72266 0.0610926 0.0272565 2024-04-22T08:30:19.344 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:19.344 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:19.403 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-22T08:30:18.207818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":551509845,"num_objects":134657,"num_object_clones":0,"num_object_copies":269314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":134657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108259,"num_write_kb":432949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3943,"num_bytes_recovered":16077417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134784,"ondisk_log_size":134784,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1857748,"kb_used_data":1095164,"kb_used_omap":12,"kb_used_meta":762547,"kb_avail":281257772,"statfs":{"total":289910292480,"available":288007958528,"internally_reserved":0,"allocated":1121447936,"data_stored":1120750588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":780848238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18702336,"num_objects":4566,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3805,"num_objects_misplaced":3464,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":18264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":1388544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.071409"},"pg_stats":[{"pgid":"2.f","version":"37'8483","reported_seq":8546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.455755+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:14.455755+0000","last_peered":"2024-04-22T08:30:14.455755+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:14.455755+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8483,"log_dups_size":0,"ondisk_log_size":8483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34746368,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":33932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4689},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8485","reported_seq":8600,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.935859+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:14.935859+0000","last_peered":"2024-04-22T08:30:14.935859+0000","last_clean":"2024-04-22T08:30:14.935859+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:14.935859+0000","last_undegraded":"2024-04-22T08:30:14.935859+0000","last_fullsized":"2024-04-22T08:30:14.935859+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34729984,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8347","reported_seq":8470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.845574+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:14.845574+0000","last_peered":"2024-04-22T08:30:14.845574+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:14.845574+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34172928,"num_objects":8343,"num_object_clones":0,"num_object_copies":16686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":20696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8436","reported_seq":8466,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.859139+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:14.859139+0000","last_peered":"2024-04-22T08:30:14.859139+0000","last_clean":"2024-04-22T08:30:14.859139+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:14.859139+0000","last_undegraded":"2024-04-22T08:30:14.859139+0000","last_fullsized":"2024-04-22T08:30:14.859139+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34553856,"num_objects":8436,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8375","reported_seq":8513,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.475537+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:14.475537+0000","last_peered":"2024-04-22T08:30:14.475537+0000","last_clean":"2024-04-22T08:30:14.475537+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:14.475537+0000","last_undegraded":"2024-04-22T08:30:14.475537+0000","last_fullsized":"2024-04-22T08:30:14.475537+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275328,"num_objects":8368,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8464","reported_seq":8626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.344798+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:14.344798+0000","last_peered":"2024-04-22T08:30:14.344798+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:14.344798+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8464,"log_dups_size":0,"ondisk_log_size":8464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34648064,"num_objects":8459,"num_object_clones":0,"num_object_copies":16918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":21332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4580},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8467","reported_seq":16203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:18.058294+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:18.058294+0000","last_peered":"2024-04-22T08:30:18.058294+0000","last_clean":"2024-04-22T08:30:18.058294+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:18.058294+0000","last_undegraded":"2024-04-22T08:30:18.058294+0000","last_fullsized":"2024-04-22T08:30:18.058294+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34656256,"num_objects":8461,"num_object_clones":0,"num_object_copies":16922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8318","reported_seq":8461,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:14.435408+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:14.435408+0000","last_peered":"2024-04-22T08:30:14.435408+0000","last_clean":"2024-04-22T08:30:14.435408+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:14.435408+0000","last_undegraded":"2024-04-22T08:30:14.435408+0000","last_fullsized":"2024-04-22T08:30:14.435408+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34062336,"num_objects":8316,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":19864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8551","reported_seq":8674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.887855+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:14.887855+0000","last_peered":"2024-04-22T08:30:14.887855+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:14.887855+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8551,"log_dups_size":0,"ondisk_log_size":8551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34759890,"num_objects":8488,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8491,"num_write_kb":33948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8430","reported_seq":8507,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.323467+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:14.323467+0000","last_peered":"2024-04-22T08:30:14.323467+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:14.323467+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34497746,"num_objects":8424,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":33692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4653},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8370","reported_seq":8516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.703641+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:14.703641+0000","last_peered":"2024-04-22T08:30:14.703641+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:14.703641+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34263040,"num_objects":8365,"num_object_clones":0,"num_object_copies":16730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":19296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8411","reported_seq":8747,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:14.914197+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:14.914197+0000","last_peered":"2024-04-22T08:30:14.914197+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:14.914197+0000","last_undegraded":"2024-04-22T08:30:14.914197+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8411,"log_dups_size":0,"ondisk_log_size":8411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34428114,"num_objects":8407,"num_object_clones":0,"num_object_copies":16814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":8407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8409,"num_write_kb":33623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3663},{"shards":"2","objects":4744}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8438","reported_seq":8517,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.453776+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:14.453776+0000","last_peered":"2024-04-22T08:30:14.453776+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:14.453776+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8438,"log_dups_size":0,"ondisk_log_size":8438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34554473,"num_objects":8437,"num_object_clones":0,"num_object_copies":16874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":33745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4716},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8304","reported_seq":8479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.428095+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:14.428095+0000","last_peered":"2024-04-22T08:30:14.428095+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:14.428095+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33980416,"num_objects":8296,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4535,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4459},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8434","reported_seq":8510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.415645+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:14.415645+0000","last_peered":"2024-04-22T08:30:14.415645+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:14.415645+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34530514,"num_objects":8432,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":33722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4727},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8471","reported_seq":8615,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:14.408874+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:14.408874+0000","last_peered":"2024-04-22T08:30:14.408874+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:14.408874+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34650532,"num_objects":8463,"num_object_clones":0,"num_object_copies":16926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":33843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4582},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":551509845,"num_objects":134657,"num_object_clones":0,"num_object_copies":269314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3663,"num_objects_unfound":0,"num_objects_dirty":134657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108259,"num_write_kb":432949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3943,"num_bytes_recovered":16077417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123532800,"data_stored":1123452783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134784,"ondisk_log_size":134784,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634424,"kb_used_data":390320,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93737416,"statfs":{"total":96636764160,"available":95987113984,"internally_reserved":0,"allocated":399687680,"data_stored":399476270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.373}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":4.2069999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920798,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457708,"kb_used_data":241508,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93914132,"statfs":{"total":96636764160,"available":96168071168,"internally_reserved":0,"allocated":247304192,"data_stored":247023202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313263,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765616,"kb_used_data":463336,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93606224,"statfs":{"total":96636764160,"available":95852773376,"internally_reserved":0,"allocated":474456064,"data_stored":474251116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":309522131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.875},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.84199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":475459584,"data_stored":475456105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248295424,"data_stored":248222365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399777792,"data_stored":399774313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'8483", 'reported_seq': 8546, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.455755+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:14.455755+0000', 'last_peered': '2024-04-22T08:30:14.455755+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:14.455755+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8483, 'log_dups_size': 0, 'ondisk_log_size': 8483, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34746368, 'num_objects': 8483, 'num_object_clones': 0, 'num_object_copies': 16966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8483, 'num_write_kb': 33932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4689}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'8347", 'reported_seq': 8470, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.845574+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:14.845574+0000', 'last_peered': '2024-04-22T08:30:14.845574+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:14.845574+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 8347, 'log_dups_size': 0, 'ondisk_log_size': 8347, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34172928, 'num_objects': 8343, 'num_object_clones': 0, 'num_object_copies': 16686, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5174, 'num_write_kb': 20696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 4588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'8464", 'reported_seq': 8626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.344798+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:14.344798+0000', 'last_peered': '2024-04-22T08:30:14.344798+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:14.344798+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 8464, 'log_dups_size': 0, 'ondisk_log_size': 8464, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34648064, 'num_objects': 8459, 'num_object_clones': 0, 'num_object_copies': 16918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5333, 'num_write_kb': 21332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4580}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'8551", 'reported_seq': 8674, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.887855+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:14.887855+0000', 'last_peered': '2024-04-22T08:30:14.887855+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:14.887855+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 8551, 'log_dups_size': 0, 'ondisk_log_size': 8551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34759890, 'num_objects': 8488, 'num_object_clones': 0, 'num_object_copies': 16976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8491, 'num_write_kb': 33948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 4600}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'8430", 'reported_seq': 8507, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.323467+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:14.323467+0000', 'last_peered': '2024-04-22T08:30:14.323467+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:14.323467+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8430, 'log_dups_size': 0, 'ondisk_log_size': 8430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34497746, 'num_objects': 8424, 'num_object_clones': 0, 'num_object_copies': 16848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8428, 'num_write_kb': 33692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4653}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'8370", 'reported_seq': 8516, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.703641+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:14.703641+0000', 'last_peered': '2024-04-22T08:30:14.703641+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:14.703641+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 8370, 'log_dups_size': 0, 'ondisk_log_size': 8370, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34263040, 'num_objects': 8365, 'num_object_clones': 0, 'num_object_copies': 16730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4824, 'num_write_kb': 19296, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 4615}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'8411", 'reported_seq': 8747, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:14.914197+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:14.914197+0000', 'last_peered': '2024-04-22T08:30:14.914197+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:14.914197+0000', 'last_undegraded': '2024-04-22T08:30:14.914197+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8411, 'log_dups_size': 0, 'ondisk_log_size': 8411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34428114, 'num_objects': 8407, 'num_object_clones': 0, 'num_object_copies': 16814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3663, 'num_objects_unfound': 0, 'num_objects_dirty': 8407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8409, 'num_write_kb': 33623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 142, 'num_bytes_recovered': 573440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3663}, {'shards': '2', 'objects': 4744}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'8438", 'reported_seq': 8517, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.453776+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:14.453776+0000', 'last_peered': '2024-04-22T08:30:14.453776+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:14.453776+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8438, 'log_dups_size': 0, 'ondisk_log_size': 8438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34554473, 'num_objects': 8437, 'num_object_clones': 0, 'num_object_copies': 16874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8437, 'num_write_kb': 33745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4716}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:19.406 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'8304", 'reported_seq': 8479, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.428095+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:14.428095+0000', 'last_peered': '2024-04-22T08:30:14.428095+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:14.428095+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8304, 'log_dups_size': 0, 'ondisk_log_size': 8304, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33980416, 'num_objects': 8296, 'num_object_clones': 0, 'num_object_copies': 16592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4535, 'num_write_kb': 18140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4459}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:19.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'8434", 'reported_seq': 8510, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.415645+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:14.415645+0000', 'last_peered': '2024-04-22T08:30:14.415645+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:14.415645+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8434, 'log_dups_size': 0, 'ondisk_log_size': 8434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34530514, 'num_objects': 8432, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8432, 'num_write_kb': 33722, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4727}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:19.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'8471", 'reported_seq': 8615, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:14.408874+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:14.408874+0000', 'last_peered': '2024-04-22T08:30:14.408874+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:14.408874+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 8471, 'log_dups_size': 0, 'ondisk_log_size': 8471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34650532, 'num_objects': 8463, 'num_object_clones': 0, 'num_object_copies': 16926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8468, 'num_write_kb': 33843, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4582}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:19.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:19.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 187 20 136944 136924 2.85916 1.67969 0.0436842 0.0273179 2024-04-22T08:30:20.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:20.285 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/269314 objects degraded (14.110%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":255,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":134657,"data_bytes":551509845,"bytes_used":1902333952,"bytes_avail":288007958528,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":269314,"degraded_ratio":0.14109552418366664,"misplaced_objects":3663,"misplaced_total":269314,"misplaced_ratio":0.013601223850226874,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":153068,"recovering_keys_per_sec":0,"num_objects_recovered":339,"num_bytes_recovered":1388544,"num_keys_recovered":0,"write_bytes_sec":2061679,"read_op_per_sec":0,"write_op_per_sec":503},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-22T08:30:02.203547+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:30:20.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:20.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:20.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 188 20 137388 137368 2.85317 1.73438 0.0130409 0.0273742 2024-04-22T08:30:21.203 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:21.205 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:21.270 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-04-22T08:30:20.208304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":560439125,"num_objects":136837,"num_object_clones":0,"num_object_copies":273674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110439,"num_write_kb":441669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136964,"ondisk_log_size":136964,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1878972,"kb_used_data":1108192,"kb_used_omap":12,"kb_used_meta":770739,"kb_avail":281236548,"statfs":{"total":289910292480,"available":287986225152,"internally_reserved":0,"allocated":1134788608,"data_stored":1134091260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":789236846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,9,7,17,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18739200,"num_objects":4575,"num_object_clones":0,"num_object_copies":9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-398,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4575,"num_write_kb":18300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.069531"},"pg_stats":[{"pgid":"2.f","version":"37'8636","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.554374+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:19.554374+0000","last_peered":"2024-04-22T08:30:19.554374+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:19.554374+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35373056,"num_objects":8636,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":34544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4842},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8645","reported_seq":8760,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.955242+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:19.955242+0000","last_peered":"2024-04-22T08:30:19.955242+0000","last_clean":"2024-04-22T08:30:19.955242+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:19.955242+0000","last_undegraded":"2024-04-22T08:30:19.955242+0000","last_fullsized":"2024-04-22T08:30:19.955242+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35385344,"num_objects":8639,"num_object_clones":0,"num_object_copies":17278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5701,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8500","reported_seq":8623,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.944562+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:19.944562+0000","last_peered":"2024-04-22T08:30:19.944562+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:19.944562+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":8496,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5327,"num_write_kb":21308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8601","reported_seq":8630,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.974752+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:19.974752+0000","last_peered":"2024-04-22T08:30:19.974752+0000","last_clean":"2024-04-22T08:30:19.974752+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:19.974752+0000","last_undegraded":"2024-04-22T08:30:19.974752+0000","last_fullsized":"2024-04-22T08:30:19.974752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35229696,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8494","reported_seq":8632,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.402788+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:19.402788+0000","last_peered":"2024-04-22T08:30:19.402788+0000","last_clean":"2024-04-22T08:30:19.402788+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:19.402788+0000","last_undegraded":"2024-04-22T08:30:19.402788+0000","last_fullsized":"2024-04-22T08:30:19.402788+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34762752,"num_objects":8487,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":22344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8596","reported_seq":8758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.441835+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:19.441835+0000","last_peered":"2024-04-22T08:30:19.441835+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:19.441835+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5465,"num_write_kb":21860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4712},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8467","reported_seq":16203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:18.058294+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:18.058294+0000","last_peered":"2024-04-22T08:30:18.058294+0000","last_clean":"2024-04-22T08:30:18.058294+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:18.058294+0000","last_undegraded":"2024-04-22T08:30:18.058294+0000","last_fullsized":"2024-04-22T08:30:18.058294+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34656256,"num_objects":8461,"num_object_clones":0,"num_object_copies":16922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8459","reported_seq":8603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.276435+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:19.276435+0000","last_peered":"2024-04-22T08:30:19.276435+0000","last_clean":"2024-04-22T08:30:19.276435+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:19.276435+0000","last_undegraded":"2024-04-22T08:30:19.276435+0000","last_fullsized":"2024-04-22T08:30:19.276435+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34639872,"num_objects":8457,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8685","reported_seq":8808,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.854881+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:19.854881+0000","last_peered":"2024-04-22T08:30:19.854881+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:19.854881+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35308754,"num_objects":8622,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":34484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4734}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8592","reported_seq":8669,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.412902+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:19.412902+0000","last_peered":"2024-04-22T08:30:19.412902+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:19.412902+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35161298,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":34340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4815},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8521","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.891382+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:19.891382+0000","last_peered":"2024-04-22T08:30:19.891382+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:19.891382+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":8516,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":19900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8541","reported_seq":9081,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:19.967395+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:19.967395+0000","last_peered":"2024-04-22T08:30:19.967395+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:19.967395+0000","last_undegraded":"2024-04-22T08:30:19.967395+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960594,"num_objects":8537,"num_object_clones":0,"num_object_copies":17074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":8537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":34143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3561},{"shards":"2","objects":4976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8602","reported_seq":8681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.443482+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:19.443482+0000","last_peered":"2024-04-22T08:30:19.443482+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:19.443482+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35226217,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4880},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8436","reported_seq":8610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.522838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:19.522838+0000","last_peered":"2024-04-22T08:30:19.522838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:19.522838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34521088,"num_objects":8428,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":18668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4591},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8574","reported_seq":8650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.530191+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:19.530191+0000","last_peered":"2024-04-22T08:30:19.530191+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:19.530191+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35103954,"num_objects":8572,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":34282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4867},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8615","reported_seq":8759,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.486455+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:19.486455+0000","last_peered":"2024-04-22T08:30:19.486455+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:19.486455+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35240356,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":34419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4726},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":560439125,"num_objects":136837,"num_object_clones":0,"num_object_copies":273674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110439,"num_write_kb":441669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1135554560,"data_stored":1135474543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136964,"ondisk_log_size":136964,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117036,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643208,"kb_used_data":395004,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93728632,"statfs":{"total":96636764160,"available":95978119168,"internally_reserved":0,"allocated":404484096,"data_stored":404272686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920798,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457708,"kb_used_data":241508,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93914132,"statfs":{"total":96636764160,"available":96168071168,"internally_reserved":0,"allocated":247304192,"data_stored":247023202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778056,"kb_used_data":471680,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93593784,"statfs":{"total":96636764160,"available":95840034816,"internally_reserved":0,"allocated":483000320,"data_stored":482795372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":313716435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":482791424,"data_stored":482787945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248295424,"data_stored":248222365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404467712,"data_stored":404464233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:21.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 189 19 137863 137844 2.84791 1.85938 0.0184277 0.0274239 2024-04-22T08:30:22.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 190 20 138347 138327 2.84286 1.88672 0.026707 0.027474 2024-04-22T08:30:23.881 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 191 20 138803 138783 2.8373 1.78125 0.0806493 0.0275271 2024-04-22T08:30:24.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:24.884 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 192 20 139270 139250 2.83198 1.82422 0.0147733 0.0275783 2024-04-22T08:30:25.218 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:25.219 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:25.260 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-22T08:30:24.209234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561028949,"num_objects":136981,"num_object_clones":0,"num_object_copies":273962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110583,"num_write_kb":442245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137108,"ondisk_log_size":137108,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1878000,"kb_used_data":1115412,"kb_used_omap":12,"kb_used_meta":762547,"kb_avail":281237520,"statfs":{"total":289910292480,"available":287987220480,"internally_reserved":0,"allocated":1142181888,"data_stored":1141484540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":780848238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,9,7,17,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19329024,"num_objects":4719,"num_object_clones":0,"num_object_copies":9438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-398,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":18876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"2.f","version":"37'8636","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.554374+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:19.554374+0000","last_peered":"2024-04-22T08:30:19.554374+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:19.554374+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35373056,"num_objects":8636,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":34544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4842},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8645","reported_seq":8760,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.955242+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:19.955242+0000","last_peered":"2024-04-22T08:30:19.955242+0000","last_clean":"2024-04-22T08:30:19.955242+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:19.955242+0000","last_undegraded":"2024-04-22T08:30:19.955242+0000","last_fullsized":"2024-04-22T08:30:19.955242+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35385344,"num_objects":8639,"num_object_clones":0,"num_object_copies":17278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5701,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8500","reported_seq":8623,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.944562+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:19.944562+0000","last_peered":"2024-04-22T08:30:19.944562+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:19.944562+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":8496,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5327,"num_write_kb":21308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8601","reported_seq":8630,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.974752+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:19.974752+0000","last_peered":"2024-04-22T08:30:19.974752+0000","last_clean":"2024-04-22T08:30:19.974752+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:19.974752+0000","last_undegraded":"2024-04-22T08:30:19.974752+0000","last_fullsized":"2024-04-22T08:30:19.974752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35229696,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8494","reported_seq":8632,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.402788+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:19.402788+0000","last_peered":"2024-04-22T08:30:19.402788+0000","last_clean":"2024-04-22T08:30:19.402788+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:19.402788+0000","last_undegraded":"2024-04-22T08:30:19.402788+0000","last_fullsized":"2024-04-22T08:30:19.402788+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34762752,"num_objects":8487,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":22344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8596","reported_seq":8758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.441835+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:19.441835+0000","last_peered":"2024-04-22T08:30:19.441835+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:19.441835+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5465,"num_write_kb":21860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4712},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8611","reported_seq":16347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:23.073848+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:23.073848+0000","last_peered":"2024-04-22T08:30:23.073848+0000","last_clean":"2024-04-22T08:30:23.073848+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:23.073848+0000","last_undegraded":"2024-04-22T08:30:23.073848+0000","last_fullsized":"2024-04-22T08:30:23.073848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35246080,"num_objects":8605,"num_object_clones":0,"num_object_copies":17210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":19916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8459","reported_seq":8603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.276435+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:19.276435+0000","last_peered":"2024-04-22T08:30:19.276435+0000","last_clean":"2024-04-22T08:30:19.276435+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:19.276435+0000","last_undegraded":"2024-04-22T08:30:19.276435+0000","last_fullsized":"2024-04-22T08:30:19.276435+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34639872,"num_objects":8457,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8685","reported_seq":8808,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.854881+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:19.854881+0000","last_peered":"2024-04-22T08:30:19.854881+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:19.854881+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35308754,"num_objects":8622,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":34484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4734}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8592","reported_seq":8669,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.412902+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:19.412902+0000","last_peered":"2024-04-22T08:30:19.412902+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:19.412902+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35161298,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":34340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4815},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8521","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.891382+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:19.891382+0000","last_peered":"2024-04-22T08:30:19.891382+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:19.891382+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":8516,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":19900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8541","reported_seq":9081,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:19.967395+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:19.967395+0000","last_peered":"2024-04-22T08:30:19.967395+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:19.967395+0000","last_undegraded":"2024-04-22T08:30:19.967395+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960594,"num_objects":8537,"num_object_clones":0,"num_object_copies":17074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":8537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":34143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3561},{"shards":"2","objects":4976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8602","reported_seq":8681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.443482+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:19.443482+0000","last_peered":"2024-04-22T08:30:19.443482+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:19.443482+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35226217,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4880},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8436","reported_seq":8610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.522838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:19.522838+0000","last_peered":"2024-04-22T08:30:19.522838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:19.522838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34521088,"num_objects":8428,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":18668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4591},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8574","reported_seq":8650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.530191+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:19.530191+0000","last_peered":"2024-04-22T08:30:19.530191+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:19.530191+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35103954,"num_objects":8572,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":34282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4867},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8615","reported_seq":8759,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.486455+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:19.486455+0000","last_peered":"2024-04-22T08:30:19.486455+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:19.486455+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35240356,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":34419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4726},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":561028949,"num_objects":136981,"num_object_clones":0,"num_object_copies":273962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110583,"num_write_kb":442245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1142939648,"data_stored":1142859631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137108,"ondisk_log_size":137108,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117036,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643208,"kb_used_data":395004,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93728632,"statfs":{"total":96636764160,"available":95978119168,"internally_reserved":0,"allocated":404484096,"data_stored":404272686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920799,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456736,"kb_used_data":248728,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93915104,"statfs":{"total":96636764160,"available":96169066496,"internally_reserved":0,"allocated":254697472,"data_stored":254416482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":212987596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778056,"kb_used_data":471680,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93593784,"statfs":{"total":96636764160,"available":95840034816,"internally_reserved":0,"allocated":483000320,"data_stored":482795372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":313716435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":482791424,"data_stored":482787945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255680512,"data_stored":255607453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404467712,"data_stored":404464233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:25.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:25.884 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 193 20 139696 139676 2.82593 1.66406 0.00911188 0.0276386 2024-04-22T08:30:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:25.991 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-22T08:30:24.209234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561028949,"num_objects":136981,"num_object_clones":0,"num_object_copies":273962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110583,"num_write_kb":442245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137108,"ondisk_log_size":137108,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1878000,"kb_used_data":1115412,"kb_used_omap":12,"kb_used_meta":762547,"kb_avail":281237520,"statfs":{"total":289910292480,"available":287987220480,"internally_reserved":0,"allocated":1142181888,"data_stored":1141484540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":780848238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,9,7,17,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19329024,"num_objects":4719,"num_object_clones":0,"num_object_copies":9438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-398,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":18876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"2.f","version":"37'8636","reported_seq":8699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.554374+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:19.554374+0000","last_peered":"2024-04-22T08:30:19.554374+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:19.554374+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35373056,"num_objects":8636,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":34544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4842},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8645","reported_seq":8760,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.955242+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:19.955242+0000","last_peered":"2024-04-22T08:30:19.955242+0000","last_clean":"2024-04-22T08:30:19.955242+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:19.955242+0000","last_undegraded":"2024-04-22T08:30:19.955242+0000","last_fullsized":"2024-04-22T08:30:19.955242+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35385344,"num_objects":8639,"num_object_clones":0,"num_object_copies":17278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5701,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8500","reported_seq":8623,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.944562+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:19.944562+0000","last_peered":"2024-04-22T08:30:19.944562+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:19.944562+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":8496,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5327,"num_write_kb":21308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8601","reported_seq":8630,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.974752+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:19.974752+0000","last_peered":"2024-04-22T08:30:19.974752+0000","last_clean":"2024-04-22T08:30:19.974752+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:19.974752+0000","last_undegraded":"2024-04-22T08:30:19.974752+0000","last_fullsized":"2024-04-22T08:30:19.974752+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35229696,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8494","reported_seq":8632,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.402788+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:19.402788+0000","last_peered":"2024-04-22T08:30:19.402788+0000","last_clean":"2024-04-22T08:30:19.402788+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:19.402788+0000","last_undegraded":"2024-04-22T08:30:19.402788+0000","last_fullsized":"2024-04-22T08:30:19.402788+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34762752,"num_objects":8487,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":22344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8596","reported_seq":8758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.441835+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:19.441835+0000","last_peered":"2024-04-22T08:30:19.441835+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:19.441835+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5465,"num_write_kb":21860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4712},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8611","reported_seq":16347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:23.073848+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:23.073848+0000","last_peered":"2024-04-22T08:30:23.073848+0000","last_clean":"2024-04-22T08:30:23.073848+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:23.073848+0000","last_undegraded":"2024-04-22T08:30:23.073848+0000","last_fullsized":"2024-04-22T08:30:23.073848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35246080,"num_objects":8605,"num_object_clones":0,"num_object_copies":17210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":19916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8459","reported_seq":8603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:19.276435+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:19.276435+0000","last_peered":"2024-04-22T08:30:19.276435+0000","last_clean":"2024-04-22T08:30:19.276435+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:19.276435+0000","last_undegraded":"2024-04-22T08:30:19.276435+0000","last_fullsized":"2024-04-22T08:30:19.276435+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34639872,"num_objects":8457,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8685","reported_seq":8808,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.854881+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:19.854881+0000","last_peered":"2024-04-22T08:30:19.854881+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:19.854881+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35308754,"num_objects":8622,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":34484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4734}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8592","reported_seq":8669,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.412902+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:19.412902+0000","last_peered":"2024-04-22T08:30:19.412902+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:19.412902+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35161298,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":34340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4815},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8521","reported_seq":8667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.891382+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:19.891382+0000","last_peered":"2024-04-22T08:30:19.891382+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:19.891382+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":8516,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":19900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8541","reported_seq":9081,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:19.967395+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:19.967395+0000","last_peered":"2024-04-22T08:30:19.967395+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:19.967395+0000","last_undegraded":"2024-04-22T08:30:19.967395+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34960594,"num_objects":8537,"num_object_clones":0,"num_object_copies":17074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":8537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":34143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3561},{"shards":"2","objects":4976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8602","reported_seq":8681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.443482+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:19.443482+0000","last_peered":"2024-04-22T08:30:19.443482+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:19.443482+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35226217,"num_objects":8601,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":34401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4880},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8436","reported_seq":8610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.522838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:19.522838+0000","last_peered":"2024-04-22T08:30:19.522838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:19.522838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34521088,"num_objects":8428,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":18668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4591},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8574","reported_seq":8650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.530191+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:19.530191+0000","last_peered":"2024-04-22T08:30:19.530191+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:19.530191+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35103954,"num_objects":8572,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":34282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4867},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8615","reported_seq":8759,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:19.486455+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:19.486455+0000","last_peered":"2024-04-22T08:30:19.486455+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:19.486455+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35240356,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":34419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4726},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":561028949,"num_objects":136981,"num_object_clones":0,"num_object_copies":273962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3561,"num_objects_unfound":0,"num_objects_dirty":136981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110583,"num_write_kb":442245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4045,"num_bytes_recovered":16495209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1142939648,"data_stored":1142859631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137108,"ondisk_log_size":137108,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117036,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643208,"kb_used_data":395004,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93728632,"statfs":{"total":96636764160,"available":95978119168,"internally_reserved":0,"allocated":404484096,"data_stored":404272686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920799,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456736,"kb_used_data":248728,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93915104,"statfs":{"total":96636764160,"available":96169066496,"internally_reserved":0,"allocated":254697472,"data_stored":254416482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":212987596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313264,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778056,"kb_used_data":471680,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93593784,"statfs":{"total":96636764160,"available":95840034816,"internally_reserved":0,"allocated":483000320,"data_stored":482795372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":313716435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":482791424,"data_stored":482787945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255680512,"data_stored":255607453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404467712,"data_stored":404464233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'8636", 'reported_seq': 8699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.554374+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:19.554374+0000', 'last_peered': '2024-04-22T08:30:19.554374+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:19.554374+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8636, 'log_dups_size': 0, 'ondisk_log_size': 8636, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35373056, 'num_objects': 8636, 'num_object_clones': 0, 'num_object_copies': 17272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8636, 'num_write_kb': 34544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4842}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'8500", 'reported_seq': 8623, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.944562+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:19.944562+0000', 'last_peered': '2024-04-22T08:30:19.944562+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:19.944562+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 8500, 'log_dups_size': 0, 'ondisk_log_size': 8500, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34799616, 'num_objects': 8496, 'num_object_clones': 0, 'num_object_copies': 16992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5327, 'num_write_kb': 21308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 4741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:25.993 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'8596", 'reported_seq': 8758, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.441835+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:19.441835+0000', 'last_peered': '2024-04-22T08:30:19.441835+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:19.441835+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 8596, 'log_dups_size': 0, 'ondisk_log_size': 8596, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35188736, 'num_objects': 8591, 'num_object_clones': 0, 'num_object_copies': 17182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5465, 'num_write_kb': 21860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4712}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'8685", 'reported_seq': 8808, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.854881+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:19.854881+0000', 'last_peered': '2024-04-22T08:30:19.854881+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:19.854881+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 8685, 'log_dups_size': 0, 'ondisk_log_size': 8685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35308754, 'num_objects': 8622, 'num_object_clones': 0, 'num_object_copies': 17244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8625, 'num_write_kb': 34484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 4734}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'8592", 'reported_seq': 8669, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.412902+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:19.412902+0000', 'last_peered': '2024-04-22T08:30:19.412902+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:19.412902+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8592, 'log_dups_size': 0, 'ondisk_log_size': 8592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35161298, 'num_objects': 8586, 'num_object_clones': 0, 'num_object_copies': 17172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8590, 'num_write_kb': 34340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4815}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'8521", 'reported_seq': 8667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.891382+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:19.891382+0000', 'last_peered': '2024-04-22T08:30:19.891382+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:19.891382+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 8521, 'log_dups_size': 0, 'ondisk_log_size': 8521, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34881536, 'num_objects': 8516, 'num_object_clones': 0, 'num_object_copies': 17032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4975, 'num_write_kb': 19900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 4766}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'8541", 'reported_seq': 9081, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:19.967395+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:19.967395+0000', 'last_peered': '2024-04-22T08:30:19.967395+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:19.967395+0000', 'last_undegraded': '2024-04-22T08:30:19.967395+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8541, 'log_dups_size': 0, 'ondisk_log_size': 8541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34960594, 'num_objects': 8537, 'num_object_clones': 0, 'num_object_copies': 17074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3561, 'num_objects_unfound': 0, 'num_objects_dirty': 8537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8539, 'num_write_kb': 34143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 244, 'num_bytes_recovered': 991232, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3561}, {'shards': '2', 'objects': 4976}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:25.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'8602", 'reported_seq': 8681, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.443482+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:19.443482+0000', 'last_peered': '2024-04-22T08:30:19.443482+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:19.443482+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8602, 'log_dups_size': 0, 'ondisk_log_size': 8602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35226217, 'num_objects': 8601, 'num_object_clones': 0, 'num_object_copies': 17202, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8601, 'num_write_kb': 34401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4880}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'8436", 'reported_seq': 8610, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.522838+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:19.522838+0000', 'last_peered': '2024-04-22T08:30:19.522838+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:19.522838+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8436, 'log_dups_size': 0, 'ondisk_log_size': 8436, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34521088, 'num_objects': 8428, 'num_object_clones': 0, 'num_object_copies': 16856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4667, 'num_write_kb': 18668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4591}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'8574", 'reported_seq': 8650, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.530191+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:19.530191+0000', 'last_peered': '2024-04-22T08:30:19.530191+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:19.530191+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8574, 'log_dups_size': 0, 'ondisk_log_size': 8574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35103954, 'num_objects': 8572, 'num_object_clones': 0, 'num_object_copies': 17144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8572, 'num_write_kb': 34282, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4867}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:25.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'8615", 'reported_seq': 8759, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:19.486455+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:19.486455+0000', 'last_peered': '2024-04-22T08:30:19.486455+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:19.486455+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 8615, 'log_dups_size': 0, 'ondisk_log_size': 8615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35240356, 'num_objects': 8607, 'num_object_clones': 0, 'num_object_copies': 17214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8612, 'num_write_kb': 34419, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4726}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:25.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:26.885 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 194 20 140137 140117 2.82024 1.72266 0.0176579 0.0276936 2024-04-22T08:30:26.918 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:26.920 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/278320 objects degraded (13.653%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":261,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":139160,"data_bytes":569954133,"bytes_used":1949302784,"bytes_avail":287960989696,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":278320,"degraded_ratio":0.13652989364759988,"misplaced_objects":3465,"misplaced_total":278320,"misplaced_ratio":0.012449698189134807,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":100668,"recovering_keys_per_sec":0,"num_objects_recovered":295,"num_bytes_recovered":1208320,"num_keys_recovered":0,"write_bytes_sec":2270682,"read_op_per_sec":0,"write_op_per_sec":554},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-22T08:30:02.203547+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:30:26.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:26.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:27.689 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:27.692 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:27.755 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-04-22T08:30:26.209719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569954133,"num_objects":139160,"num_object_clones":0,"num_object_copies":278320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3465,"num_objects_unfound":0,"num_objects_dirty":139160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112762,"num_write_kb":450961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4141,"num_bytes_recovered":16888425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139287,"ondisk_log_size":139287,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1903616,"kb_used_data":1127080,"kb_used_omap":12,"kb_used_meta":776499,"kb_avail":281211904,"statfs":{"total":289910292480,"available":287960989696,"internally_reserved":0,"allocated":1154129920,"data_stored":1153432572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":795135086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,3,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27254784,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-295,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":26616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1208320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002904"},"pg_stats":[{"pgid":"2.f","version":"37'8772","reported_seq":8835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.594967+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:24.594967+0000","last_peered":"2024-04-22T08:30:24.594967+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:24.594967+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35930112,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":35088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4978},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8783","reported_seq":8896,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:24.972095+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:24.972095+0000","last_peered":"2024-04-22T08:30:24.972095+0000","last_clean":"2024-04-22T08:30:24.972095+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:24.972095+0000","last_undegraded":"2024-04-22T08:30:24.972095+0000","last_fullsized":"2024-04-22T08:30:24.972095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35950592,"num_objects":8777,"num_object_clones":0,"num_object_copies":17554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5839,"num_write_kb":23356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8640","reported_seq":8763,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.988946+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:24.988946+0000","last_peered":"2024-04-22T08:30:24.988946+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:24.988946+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8640,"log_dups_size":0,"ondisk_log_size":8640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35373056,"num_objects":8636,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":4881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8762","reported_seq":8792,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:24.830418+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:24.830418+0000","last_peered":"2024-04-22T08:30:24.830418+0000","last_clean":"2024-04-22T08:30:24.830418+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:24.830418+0000","last_undegraded":"2024-04-22T08:30:24.830418+0000","last_fullsized":"2024-04-22T08:30:24.830418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35889152,"num_objects":8762,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":35048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8665","reported_seq":8803,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:24.593589+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:24.593589+0000","last_peered":"2024-04-22T08:30:24.593589+0000","last_clean":"2024-04-22T08:30:24.593589+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:24.593589+0000","last_undegraded":"2024-04-22T08:30:24.593589+0000","last_fullsized":"2024-04-22T08:30:24.593589+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":8658,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5757,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8728","reported_seq":8890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.526733+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:24.526733+0000","last_peered":"2024-04-22T08:30:24.526733+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:24.526733+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35729408,"num_objects":8723,"num_object_clones":0,"num_object_copies":17446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5597,"num_write_kb":22388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4844},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8611","reported_seq":16347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:23.073848+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:23.073848+0000","last_peered":"2024-04-22T08:30:23.073848+0000","last_clean":"2024-04-22T08:30:23.073848+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:23.073848+0000","last_undegraded":"2024-04-22T08:30:23.073848+0000","last_fullsized":"2024-04-22T08:30:23.073848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35246080,"num_objects":8605,"num_object_clones":0,"num_object_copies":17210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":19916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8601","reported_seq":8740,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:24.531478+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:24.531478+0000","last_peered":"2024-04-22T08:30:24.531478+0000","last_clean":"2024-04-22T08:30:24.531478+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:24.531478+0000","last_undegraded":"2024-04-22T08:30:24.531478+0000","last_fullsized":"2024-04-22T08:30:24.531478+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35221504,"num_objects":8599,"num_object_clones":0,"num_object_copies":17198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5249,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8840","reported_seq":8963,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.828860+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:24.828860+0000","last_peered":"2024-04-22T08:30:24.828860+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:24.828860+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35943634,"num_objects":8777,"num_object_clones":0,"num_object_copies":17554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":35104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":4889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8745","reported_seq":8822,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.535252+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:24.535252+0000","last_peered":"2024-04-22T08:30:24.535252+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:24.535252+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8745,"log_dups_size":0,"ondisk_log_size":8745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35787986,"num_objects":8739,"num_object_clones":0,"num_object_copies":17478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8743,"num_write_kb":34952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4968},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8645","reported_seq":8791,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.974307+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:24.974307+0000","last_peered":"2024-04-22T08:30:24.974307+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:24.974307+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389440,"num_objects":8640,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":20396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":4890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8670","reported_seq":9402,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:24.972668+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:24.972668+0000","last_peered":"2024-04-22T08:30:24.972668+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:24.972668+0000","last_undegraded":"2024-04-22T08:30:24.972668+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35488978,"num_objects":8666,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3465,"num_objects_unfound":0,"num_objects_dirty":8666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8668,"num_write_kb":34659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3465},{"shards":"2","objects":5201}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8744","reported_seq":8823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.551639+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:24.551639+0000","last_peered":"2024-04-22T08:30:24.551639+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:24.551639+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8744,"log_dups_size":0,"ondisk_log_size":8744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35807849,"num_objects":8743,"num_object_clones":0,"num_object_copies":17486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8743,"num_write_kb":34969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5022},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8594","reported_seq":8769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.616974+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:24.616974+0000","last_peered":"2024-04-22T08:30:24.616974+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:24.616974+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8594,"log_dups_size":0,"ondisk_log_size":8594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4825,"num_write_kb":19300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4749},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8733","reported_seq":8809,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.589292+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:24.589292+0000","last_peered":"2024-04-22T08:30:24.589292+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:24.589292+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35755218,"num_objects":8731,"num_object_clones":0,"num_object_copies":17462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":34918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5026},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8754","reported_seq":8898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:24.362479+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:24.362479+0000","last_peered":"2024-04-22T08:30:24.362479+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:24.362479+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35809700,"num_objects":8746,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":34975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4865},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":569954133,"num_objects":139160,"num_object_clones":0,"num_object_copies":278320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3465,"num_objects_unfound":0,"num_objects_dirty":139160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112762,"num_write_kb":450961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4141,"num_bytes_recovered":16888425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155039232,"data_stored":1154959215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139287,"ondisk_log_size":139287,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669932,"kb_used_data":399588,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93701908,"statfs":{"total":96636764160,"available":95950753792,"internally_reserved":0,"allocated":409178112,"data_stored":408966702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":13.635},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":13.721}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":13.752000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.26400000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920799,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456736,"kb_used_data":248728,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93915104,"statfs":{"total":96636764160,"available":96169066496,"internally_reserved":0,"allocated":254697472,"data_stored":254416482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":212987596},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":4.4379999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.247}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313265,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776948,"kb_used_data":478764,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93594892,"statfs":{"total":96636764160,"available":95841169408,"internally_reserved":0,"allocated":490254336,"data_stored":490049388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":305327827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":9.3770000000000007},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.28499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":490143744,"data_stored":490140265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255680512,"data_stored":255607453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409214976,"data_stored":409211497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:27.885 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 195 20 140580 140560 2.81466 1.73047 0.023674 0.0277497 2024-04-22T08:30:28.885 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 196 20 141053 141033 2.80972 1.84766 0.0607414 0.0277971 2024-04-22T08:30:29.885 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 197 20 141527 141507 2.80486 1.85156 0.0897146 0.0278468 2024-04-22T08:30:30.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:30.885 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 198 20 141985 141965 2.79973 1.78906 0.0119239 0.0278962 2024-04-22T08:30:31.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:31.481 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-22T08:30:30.214242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579313493,"num_objects":141445,"num_object_clones":0,"num_object_copies":282890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115047,"num_write_kb":460101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141572,"ondisk_log_size":141572,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1938356,"kb_used_data":1145436,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281177164,"statfs":{"total":289910292480,"available":287925415936,"internally_reserved":0,"allocated":1172926464,"data_stored":1172229116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":811912302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,9,14,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27803648,"num_objects":6788,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-284,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006424"},"pg_stats":[{"pgid":"2.f","version":"37'8922","reported_seq":8985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.649382+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:29.649382+0000","last_peered":"2024-04-22T08:30:29.649382+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:29.649382+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36544512,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":35688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5128},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8912","reported_seq":9027,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.916367+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:29.916367+0000","last_peered":"2024-04-22T08:30:29.916367+0000","last_clean":"2024-04-22T08:30:29.916367+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:29.916367+0000","last_undegraded":"2024-04-22T08:30:29.916367+0000","last_fullsized":"2024-04-22T08:30:29.916367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36478976,"num_objects":8906,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":23872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8776","reported_seq":8899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.927649+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:29.927649+0000","last_peered":"2024-04-22T08:30:29.927649+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:29.927649+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8776,"log_dups_size":0,"ondisk_log_size":8776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35930112,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":22412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8890","reported_seq":8920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:30.005918+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:30.005918+0000","last_peered":"2024-04-22T08:30:30.005918+0000","last_clean":"2024-04-22T08:30:30.005918+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:30.005918+0000","last_undegraded":"2024-04-22T08:30:30.005918+0000","last_fullsized":"2024-04-22T08:30:30.005918+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8805","reported_seq":8942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.626480+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:29.626480+0000","last_peered":"2024-04-22T08:30:29.626480+0000","last_clean":"2024-04-22T08:30:29.626480+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:29.626480+0000","last_undegraded":"2024-04-22T08:30:29.626480+0000","last_fullsized":"2024-04-22T08:30:29.626480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36036608,"num_objects":8798,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8869","reported_seq":9030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.588528+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:29.588528+0000","last_peered":"2024-04-22T08:30:29.588528+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:29.588528+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":8864,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":22952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4985},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8754","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:28.256361+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:28.256361+0000","last_peered":"2024-04-22T08:30:28.256361+0000","last_clean":"2024-04-22T08:30:28.256361+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:28.256361+0000","last_undegraded":"2024-04-22T08:30:28.256361+0000","last_fullsized":"2024-04-22T08:30:28.256361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35831808,"num_objects":8748,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8762","reported_seq":8906,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.615465+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:29.615465+0000","last_peered":"2024-04-22T08:30:29.615465+0000","last_clean":"2024-04-22T08:30:29.615465+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:29.615465+0000","last_undegraded":"2024-04-22T08:30:29.615465+0000","last_fullsized":"2024-04-22T08:30:29.615465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35880960,"num_objects":8760,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8989","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.899289+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:29.899289+0000","last_peered":"2024-04-22T08:30:29.899289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:29.899289+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36553938,"num_objects":8926,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":35700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8909","reported_seq":8986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.523925+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:29.523925+0000","last_peered":"2024-04-22T08:30:29.523925+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:29.523925+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36459730,"num_objects":8903,"num_object_clones":0,"num_object_copies":17806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5132},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8807","reported_seq":8953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.993517+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:29.993517+0000","last_peered":"2024-04-22T08:30:29.993517+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:29.993517+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36052992,"num_objects":8802,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":21044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8814","reported_seq":9718,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:29.989806+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:29.989806+0000","last_peered":"2024-04-22T08:30:29.989806+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:29.989806+0000","last_undegraded":"2024-04-22T08:30:29.989806+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36078802,"num_objects":8810,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":8810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":35235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3379},{"shards":"2","objects":5431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8891","reported_seq":8970,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.606008+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:29.606008+0000","last_peered":"2024-04-22T08:30:29.606008+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:29.606008+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36409961,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5169},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8723","reported_seq":8898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.593307+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:29.593307+0000","last_peered":"2024-04-22T08:30:29.593307+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:29.593307+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4878},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8864","reported_seq":8940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.625950+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:29.625950+0000","last_peered":"2024-04-22T08:30:29.625950+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:29.625950+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36291794,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":35442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5157},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8885","reported_seq":9028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.603618+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:29.603618+0000","last_peered":"2024-04-22T08:30:29.603618+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:29.603618+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36346276,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":35499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579313493,"num_objects":141445,"num_object_clones":0,"num_object_copies":282890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115047,"num_write_kb":460101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1174020096,"data_stored":1173940079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141572,"ondisk_log_size":141572,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678516,"kb_used_data":404076,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93693324,"statfs":{"total":96636764160,"available":95941963776,"internally_reserved":0,"allocated":413773824,"data_stored":413562414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.502}]}]},{"osd":1,"up_from":33,"seq":141733920800,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472216,"kb_used_data":256016,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93899624,"statfs":{"total":96636764160,"available":96153214976,"internally_reserved":0,"allocated":262160384,"data_stored":261879394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.26900000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313266,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787624,"kb_used_data":485344,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93584216,"statfs":{"total":96636764160,"available":95830237184,"internally_reserved":0,"allocated":496992256,"data_stored":496787308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":309522131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.193},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.221},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":497041408,"data_stored":497037929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263024640,"data_stored":262951581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413954048,"data_stored":413950569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:31.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:31.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 199 20 142420 142400 2.79417 1.69922 0.0142419 0.027952 2024-04-22T08:30:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:32.480 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-04-22T08:30:32.214766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579313493,"num_objects":141445,"num_object_clones":0,"num_object_copies":282890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115047,"num_write_kb":460101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141572,"ondisk_log_size":141572,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1938356,"kb_used_data":1145436,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281177164,"statfs":{"total":289910292480,"available":287925415936,"internally_reserved":0,"allocated":1172926464,"data_stored":1172229116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":811912302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,9,14,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18874368,"num_objects":4608,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-182,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4608,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006462"},"pg_stats":[{"pgid":"2.f","version":"37'8922","reported_seq":8985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.649382+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:29.649382+0000","last_peered":"2024-04-22T08:30:29.649382+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:29.649382+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36544512,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":35688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5128},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8912","reported_seq":9027,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.916367+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:29.916367+0000","last_peered":"2024-04-22T08:30:29.916367+0000","last_clean":"2024-04-22T08:30:29.916367+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:29.916367+0000","last_undegraded":"2024-04-22T08:30:29.916367+0000","last_fullsized":"2024-04-22T08:30:29.916367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36478976,"num_objects":8906,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":23872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8776","reported_seq":8899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.927649+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:29.927649+0000","last_peered":"2024-04-22T08:30:29.927649+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:29.927649+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8776,"log_dups_size":0,"ondisk_log_size":8776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35930112,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":22412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8890","reported_seq":8920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:30.005918+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:30.005918+0000","last_peered":"2024-04-22T08:30:30.005918+0000","last_clean":"2024-04-22T08:30:30.005918+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:30.005918+0000","last_undegraded":"2024-04-22T08:30:30.005918+0000","last_fullsized":"2024-04-22T08:30:30.005918+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8805","reported_seq":8942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.626480+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:29.626480+0000","last_peered":"2024-04-22T08:30:29.626480+0000","last_clean":"2024-04-22T08:30:29.626480+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:29.626480+0000","last_undegraded":"2024-04-22T08:30:29.626480+0000","last_fullsized":"2024-04-22T08:30:29.626480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36036608,"num_objects":8798,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8869","reported_seq":9030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.588528+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:29.588528+0000","last_peered":"2024-04-22T08:30:29.588528+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:29.588528+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":8864,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":22952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4985},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8754","reported_seq":16487,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:28.256361+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:28.256361+0000","last_peered":"2024-04-22T08:30:28.256361+0000","last_clean":"2024-04-22T08:30:28.256361+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:28.256361+0000","last_undegraded":"2024-04-22T08:30:28.256361+0000","last_fullsized":"2024-04-22T08:30:28.256361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35831808,"num_objects":8748,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8762","reported_seq":8906,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.615465+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:29.615465+0000","last_peered":"2024-04-22T08:30:29.615465+0000","last_clean":"2024-04-22T08:30:29.615465+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:29.615465+0000","last_undegraded":"2024-04-22T08:30:29.615465+0000","last_fullsized":"2024-04-22T08:30:29.615465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35880960,"num_objects":8760,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8989","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.899289+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:29.899289+0000","last_peered":"2024-04-22T08:30:29.899289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:29.899289+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36553938,"num_objects":8926,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":35700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8909","reported_seq":8986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.523925+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:29.523925+0000","last_peered":"2024-04-22T08:30:29.523925+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:29.523925+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36459730,"num_objects":8903,"num_object_clones":0,"num_object_copies":17806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5132},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8807","reported_seq":8953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.993517+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:29.993517+0000","last_peered":"2024-04-22T08:30:29.993517+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:29.993517+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36052992,"num_objects":8802,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":21044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8814","reported_seq":9718,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:29.989806+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:29.989806+0000","last_peered":"2024-04-22T08:30:29.989806+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:29.989806+0000","last_undegraded":"2024-04-22T08:30:29.989806+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36078802,"num_objects":8810,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":8810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":35235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3379},{"shards":"2","objects":5431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8891","reported_seq":8970,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.606008+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:29.606008+0000","last_peered":"2024-04-22T08:30:29.606008+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:29.606008+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36409961,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5169},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8723","reported_seq":8898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.593307+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:29.593307+0000","last_peered":"2024-04-22T08:30:29.593307+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:29.593307+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4878},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8864","reported_seq":8940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.625950+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:29.625950+0000","last_peered":"2024-04-22T08:30:29.625950+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:29.625950+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36291794,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":35442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5157},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8885","reported_seq":9028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.603618+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:29.603618+0000","last_peered":"2024-04-22T08:30:29.603618+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:29.603618+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36346276,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":35499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579313493,"num_objects":141445,"num_object_clones":0,"num_object_copies":282890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115047,"num_write_kb":460101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1174020096,"data_stored":1173940079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141572,"ondisk_log_size":141572,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678516,"kb_used_data":404076,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93693324,"statfs":{"total":96636764160,"available":95941963776,"internally_reserved":0,"allocated":413773824,"data_stored":413562414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.502}]}]},{"osd":1,"up_from":33,"seq":141733920800,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472216,"kb_used_data":256016,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93899624,"statfs":{"total":96636764160,"available":96153214976,"internally_reserved":0,"allocated":262160384,"data_stored":261879394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":221376204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.26900000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313266,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787624,"kb_used_data":485344,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93584216,"statfs":{"total":96636764160,"available":95830237184,"internally_reserved":0,"allocated":496992256,"data_stored":496787308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":309522131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.193},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.221},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":497041408,"data_stored":497037929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263024640,"data_stored":262951581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413954048,"data_stored":413950569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:32.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:32.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'8922", 'reported_seq': 8985, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.649382+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:29.649382+0000', 'last_peered': '2024-04-22T08:30:29.649382+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:29.649382+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8922, 'log_dups_size': 0, 'ondisk_log_size': 8922, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36544512, 'num_objects': 8922, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8922, 'num_write_kb': 35688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5128}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:32.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'8776", 'reported_seq': 8899, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.927649+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:29.927649+0000', 'last_peered': '2024-04-22T08:30:29.927649+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:29.927649+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 8776, 'log_dups_size': 0, 'ondisk_log_size': 8776, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35930112, 'num_objects': 8772, 'num_object_clones': 0, 'num_object_copies': 17544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5603, 'num_write_kb': 22412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'8869", 'reported_seq': 9030, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.588528+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:29.588528+0000', 'last_peered': '2024-04-22T08:30:29.588528+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:29.588528+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 8869, 'log_dups_size': 0, 'ondisk_log_size': 8869, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36306944, 'num_objects': 8864, 'num_object_clones': 0, 'num_object_copies': 17728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5738, 'num_write_kb': 22952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4985}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'8989", 'reported_seq': 9112, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.899289+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:29.899289+0000', 'last_peered': '2024-04-22T08:30:29.899289+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:29.899289+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 8989, 'log_dups_size': 0, 'ondisk_log_size': 8989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36553938, 'num_objects': 8926, 'num_object_clones': 0, 'num_object_copies': 17852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8929, 'num_write_kb': 35700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'8909", 'reported_seq': 8986, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.523925+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:29.523925+0000', 'last_peered': '2024-04-22T08:30:29.523925+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:29.523925+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8909, 'log_dups_size': 0, 'ondisk_log_size': 8909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36459730, 'num_objects': 8903, 'num_object_clones': 0, 'num_object_copies': 17806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8907, 'num_write_kb': 35608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5132}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'8807", 'reported_seq': 8953, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.993517+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:29.993517+0000', 'last_peered': '2024-04-22T08:30:29.993517+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:29.993517+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 8807, 'log_dups_size': 0, 'ondisk_log_size': 8807, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36052992, 'num_objects': 8802, 'num_object_clones': 0, 'num_object_copies': 17604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5261, 'num_write_kb': 21044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5052}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:32.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'8814", 'reported_seq': 9718, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:29.989806+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:29.989806+0000', 'last_peered': '2024-04-22T08:30:29.989806+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:29.989806+0000', 'last_undegraded': '2024-04-22T08:30:29.989806+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8814, 'log_dups_size': 0, 'ondisk_log_size': 8814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36078802, 'num_objects': 8810, 'num_object_clones': 0, 'num_object_copies': 17620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3379, 'num_objects_unfound': 0, 'num_objects_dirty': 8810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8812, 'num_write_kb': 35235, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 426, 'num_bytes_recovered': 1736704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3379}, {'shards': '2', 'objects': 5431}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'8891", 'reported_seq': 8970, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.606008+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:29.606008+0000', 'last_peered': '2024-04-22T08:30:29.606008+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:29.606008+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8891, 'log_dups_size': 0, 'ondisk_log_size': 8891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36409961, 'num_objects': 8890, 'num_object_clones': 0, 'num_object_copies': 17780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8890, 'num_write_kb': 35557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5169}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'8723", 'reported_seq': 8898, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.593307+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:29.593307+0000', 'last_peered': '2024-04-22T08:30:29.593307+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:29.593307+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8723, 'log_dups_size': 0, 'ondisk_log_size': 8723, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35696640, 'num_objects': 8715, 'num_object_clones': 0, 'num_object_copies': 17430, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4954, 'num_write_kb': 19816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4878}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'8864", 'reported_seq': 8940, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.625950+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:29.625950+0000', 'last_peered': '2024-04-22T08:30:29.625950+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:29.625950+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8864, 'log_dups_size': 0, 'ondisk_log_size': 8864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36291794, 'num_objects': 8862, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8862, 'num_write_kb': 35442, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5157}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:32.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'8885", 'reported_seq': 9028, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:29.603618+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:29.603618+0000', 'last_peered': '2024-04-22T08:30:29.603618+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:29.603618+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 8885, 'log_dups_size': 0, 'ondisk_log_size': 8885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36346276, 'num_objects': 8877, 'num_object_clones': 0, 'num_object_copies': 17754, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8882, 'num_write_kb': 35499, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4996}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:32.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:32.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:30:32.889185+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.0280052 2024-04-22T08:30:32.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:30:32.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 200 20 142863 142843 2.78886 1.73047 0.0625325 0.0280052 2024-04-22T08:30:33.631 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:33.632 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/282890 objects degraded (13.432%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":268,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":141445,"data_bytes":579313493,"bytes_used":1984876544,"bytes_avail":287925415936,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":282890,"degraded_ratio":0.13432429566262505,"misplaced_objects":3379,"misplaced_total":282890,"misplaced_ratio":0.011944572095160665,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":62089,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":745472,"num_keys_recovered":0,"write_bytes_sec":1572017,"read_op_per_sec":0,"write_op_per_sec":383},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:30:33.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:33.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:33.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 201 20 143286 143266 2.7832 1.65234 0.0920475 0.0280643 2024-04-22T08:30:34.553 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:34.553 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:34.622 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-04-22T08:30:34.215294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579825493,"num_objects":141570,"num_object_clones":0,"num_object_copies":283140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115172,"num_write_kb":460601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141697,"ondisk_log_size":141697,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1950736,"kb_used_data":1153720,"kb_used_omap":12,"kb_used_meta":796979,"kb_avail":281164784,"statfs":{"total":289910292480,"available":287912738816,"internally_reserved":0,"allocated":1181409280,"data_stored":1180711932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":816106606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,9,18,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19386368,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-182,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":18932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006465"},"pg_stats":[{"pgid":"2.f","version":"37'8922","reported_seq":8985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.649382+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:29.649382+0000","last_peered":"2024-04-22T08:30:29.649382+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:29.649382+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36544512,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":35688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5128},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'8912","reported_seq":9027,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.916367+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:29.916367+0000","last_peered":"2024-04-22T08:30:29.916367+0000","last_clean":"2024-04-22T08:30:29.916367+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:29.916367+0000","last_undegraded":"2024-04-22T08:30:29.916367+0000","last_fullsized":"2024-04-22T08:30:29.916367+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36478976,"num_objects":8906,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":23872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8776","reported_seq":8899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.927649+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:29.927649+0000","last_peered":"2024-04-22T08:30:29.927649+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:29.927649+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8776,"log_dups_size":0,"ondisk_log_size":8776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35930112,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5603,"num_write_kb":22412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'8890","reported_seq":8920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:30.005918+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:30.005918+0000","last_peered":"2024-04-22T08:30:30.005918+0000","last_clean":"2024-04-22T08:30:30.005918+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:30.005918+0000","last_undegraded":"2024-04-22T08:30:30.005918+0000","last_fullsized":"2024-04-22T08:30:30.005918+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8805","reported_seq":8942,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.626480+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:29.626480+0000","last_peered":"2024-04-22T08:30:29.626480+0000","last_clean":"2024-04-22T08:30:29.626480+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:29.626480+0000","last_undegraded":"2024-04-22T08:30:29.626480+0000","last_fullsized":"2024-04-22T08:30:29.626480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36036608,"num_objects":8798,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'8869","reported_seq":9030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.588528+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:29.588528+0000","last_peered":"2024-04-22T08:30:29.588528+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:29.588528+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":8864,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":22952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4985},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8879","reported_seq":16614,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:33.302083+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:33.302083+0000","last_peered":"2024-04-22T08:30:33.302083+0000","last_clean":"2024-04-22T08:30:33.302083+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:33.302083+0000","last_undegraded":"2024-04-22T08:30:33.302083+0000","last_fullsized":"2024-04-22T08:30:33.302083+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36343808,"num_objects":8873,"num_object_clones":0,"num_object_copies":17746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8762","reported_seq":8906,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:29.615465+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:29.615465+0000","last_peered":"2024-04-22T08:30:29.615465+0000","last_clean":"2024-04-22T08:30:29.615465+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:29.615465+0000","last_undegraded":"2024-04-22T08:30:29.615465+0000","last_fullsized":"2024-04-22T08:30:29.615465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35880960,"num_objects":8760,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'8989","reported_seq":9112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.899289+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:29.899289+0000","last_peered":"2024-04-22T08:30:29.899289+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:29.899289+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36553938,"num_objects":8926,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":35700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'8909","reported_seq":8986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.523925+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:29.523925+0000","last_peered":"2024-04-22T08:30:29.523925+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:29.523925+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36459730,"num_objects":8903,"num_object_clones":0,"num_object_copies":17806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5132},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8807","reported_seq":8953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.993517+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:29.993517+0000","last_peered":"2024-04-22T08:30:29.993517+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:29.993517+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36052992,"num_objects":8802,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":21044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5052}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8814","reported_seq":9718,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:29.989806+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:29.989806+0000","last_peered":"2024-04-22T08:30:29.989806+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:29.989806+0000","last_undegraded":"2024-04-22T08:30:29.989806+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36078802,"num_objects":8810,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":8810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":35235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3379},{"shards":"2","objects":5431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'8891","reported_seq":8970,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.606008+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:29.606008+0000","last_peered":"2024-04-22T08:30:29.606008+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:29.606008+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36409961,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":35557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5169},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8723","reported_seq":8898,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.593307+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:29.593307+0000","last_peered":"2024-04-22T08:30:29.593307+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:29.593307+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":19816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4878},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'8864","reported_seq":8940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.625950+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:29.625950+0000","last_peered":"2024-04-22T08:30:29.625950+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:29.625950+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36291794,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":35442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5157},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'8885","reported_seq":9028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:29.603618+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:29.603618+0000","last_peered":"2024-04-22T08:30:29.603618+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:29.603618+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36346276,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":35499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579825493,"num_objects":141570,"num_object_clones":0,"num_object_copies":283140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3379,"num_objects_unfound":0,"num_objects_dirty":141570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115172,"num_write_kb":460601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4227,"num_bytes_recovered":17240681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181421568,"data_stored":1181341551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141697,"ondisk_log_size":141697,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678516,"kb_used_data":404076,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93693324,"statfs":{"total":96636764160,"available":95941963776,"internally_reserved":0,"allocated":413773824,"data_stored":413562414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.502}]}]},{"osd":1,"up_from":33,"seq":141733920801,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484596,"kb_used_data":264300,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93887244,"statfs":{"total":96636764160,"available":96140537856,"internally_reserved":0,"allocated":270643200,"data_stored":270362210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":225570508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.254}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.78},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.34799999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313266,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":787624,"kb_used_data":485344,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93584216,"statfs":{"total":96636764160,"available":95830237184,"internally_reserved":0,"allocated":496992256,"data_stored":496787308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":309522131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.193},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.221},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":497041408,"data_stored":497037929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270426112,"data_stored":270353053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413954048,"data_stored":413950569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:34.891 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 202 19 143736 143717 2.77815 1.76172 0.0228014 0.028114 2024-04-22T08:30:35.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 203 20 144202 144182 2.77341 1.81641 0.0261362 0.0281618 2024-04-22T08:30:36.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 204 20 144641 144621 2.76822 1.71484 0.0764446 0.0282147 2024-04-22T08:30:37.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:37.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 205 20 145104 145084 2.76354 1.80859 0.0658736 0.0282613 2024-04-22T08:30:38.362 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:38.362 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:38.380 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T08:30:38.217834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588427093,"num_objects":143670,"num_object_clones":0,"num_object_copies":287340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":143670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117272,"num_write_kb":469001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4322,"num_bytes_recovered":17629801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143797,"ondisk_log_size":143797,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1957664,"kb_used_data":1164744,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281157856,"statfs":{"total":289910292480,"available":287905644544,"internally_reserved":0,"allocated":1192697856,"data_stored":1192000508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":811912302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,6,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18472960,"num_objects":4510,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-181,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":741376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008114"},"pg_stats":[{"pgid":"2.f","version":"37'9065","reported_seq":9128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.650852+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:34.650852+0000","last_peered":"2024-04-22T08:30:34.650852+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:34.650852+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37130240,"num_objects":9065,"num_object_clones":0,"num_object_copies":18130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9065,"num_write_kb":36260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5271},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9078","reported_seq":9193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:35.023201+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:35.023201+0000","last_peered":"2024-04-22T08:30:35.023201+0000","last_clean":"2024-04-22T08:30:35.023201+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:35.023201+0000","last_undegraded":"2024-04-22T08:30:35.023201+0000","last_fullsized":"2024-04-22T08:30:35.023201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9078,"log_dups_size":0,"ondisk_log_size":9078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37158912,"num_objects":9072,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":24536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8913","reported_seq":9036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:35.014844+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:35.014844+0000","last_peered":"2024-04-22T08:30:35.014844+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:35.014844+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36491264,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":22960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9027","reported_seq":9057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.987685+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:34.987685+0000","last_peered":"2024-04-22T08:30:34.987685+0000","last_clean":"2024-04-22T08:30:34.987685+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:34.987685+0000","last_undegraded":"2024-04-22T08:30:34.987685+0000","last_fullsized":"2024-04-22T08:30:34.987685+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36974592,"num_objects":9027,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":36108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8935","reported_seq":9073,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.694972+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:34.694972+0000","last_peered":"2024-04-22T08:30:34.694972+0000","last_clean":"2024-04-22T08:30:34.694972+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:34.694972+0000","last_undegraded":"2024-04-22T08:30:34.694972+0000","last_fullsized":"2024-04-22T08:30:34.694972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8935,"log_dups_size":0,"ondisk_log_size":8935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":8928,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9009","reported_seq":9171,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.646811+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:34.646811+0000","last_peered":"2024-04-22T08:30:34.646811+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:34.646811+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36880384,"num_objects":9004,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":23512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5125},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8879","reported_seq":16614,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:33.302083+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:33.302083+0000","last_peered":"2024-04-22T08:30:33.302083+0000","last_clean":"2024-04-22T08:30:33.302083+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:33.302083+0000","last_undegraded":"2024-04-22T08:30:33.302083+0000","last_fullsized":"2024-04-22T08:30:33.302083+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36343808,"num_objects":8873,"num_object_clones":0,"num_object_copies":17746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8899","reported_seq":9043,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.630482+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:34.630482+0000","last_peered":"2024-04-22T08:30:34.630482+0000","last_clean":"2024-04-22T08:30:34.630482+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:34.630482+0000","last_undegraded":"2024-04-22T08:30:34.630482+0000","last_fullsized":"2024-04-22T08:30:34.630482+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36442112,"num_objects":8897,"num_object_clones":0,"num_object_copies":17794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":22188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9137","reported_seq":9260,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.959823+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:34.959823+0000","last_peered":"2024-04-22T08:30:34.959823+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:34.959823+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37160146,"num_objects":9074,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":36292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9045","reported_seq":9122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.643879+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:34.643879+0000","last_peered":"2024-04-22T08:30:34.643879+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:34.643879+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016786,"num_objects":9039,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":36152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8949","reported_seq":9095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.982652+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:34.982652+0000","last_peered":"2024-04-22T08:30:34.982652+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:34.982652+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":8944,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":21612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8961","reported_seq":10052,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:35.037023+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:35.037023+0000","last_peered":"2024-04-22T08:30:35.037023+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:35.037023+0000","last_undegraded":"2024-04-22T08:30:35.037023+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680914,"num_objects":8957,"num_object_clones":0,"num_object_copies":17914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":8957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":35823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":521,"num_bytes_recovered":2125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3284},{"shards":"2","objects":5673}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9009","reported_seq":9088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.582990+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:34.582990+0000","last_peered":"2024-04-22T08:30:34.582990+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:34.582990+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36893289,"num_objects":9008,"num_object_clones":0,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":36029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8863","reported_seq":9038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.653162+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:34.653162+0000","last_peered":"2024-04-22T08:30:34.653162+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:34.653162+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8863,"log_dups_size":0,"ondisk_log_size":8863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36270080,"num_objects":8855,"num_object_clones":0,"num_object_copies":17710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":20376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5018},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9007","reported_seq":9083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.632709+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:34.632709+0000","last_peered":"2024-04-22T08:30:34.632709+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:34.632709+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9007,"log_dups_size":0,"ondisk_log_size":9007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36877522,"num_objects":9005,"num_object_clones":0,"num_object_copies":18010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9005,"num_write_kb":36014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5300},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9021","reported_seq":9165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.673809+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:34.673809+0000","last_peered":"2024-04-22T08:30:34.673809+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:34.673809+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9021,"log_dups_size":0,"ondisk_log_size":9021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36903332,"num_objects":9013,"num_object_clones":0,"num_object_copies":18026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":36043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5132},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":588427093,"num_objects":143670,"num_object_clones":0,"num_object_copies":287340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":143670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117272,"num_write_kb":469001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4322,"num_bytes_recovered":17629801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1192837120,"data_stored":1192757103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143797,"ondisk_log_size":143797,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117039,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670688,"kb_used_data":408536,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93701152,"statfs":{"total":96636764160,"available":95949979648,"internally_reserved":0,"allocated":418340864,"data_stored":418129454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.437}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.4470000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.373}]}]},{"osd":1,"up_from":33,"seq":141733920801,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484596,"kb_used_data":264300,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93887244,"statfs":{"total":96636764160,"available":96140537856,"internally_reserved":0,"allocated":270643200,"data_stored":270362210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":225570508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.254}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.78},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.34799999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313267,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802380,"kb_used_data":491908,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93569460,"statfs":{"total":96636764160,"available":95815127040,"internally_reserved":0,"allocated":503713792,"data_stored":503508844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":317910739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.371},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.403}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":503779328,"data_stored":503775849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270426112,"data_stored":270353053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418631680,"data_stored":418628201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:38.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:38.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 206 20 145500 145480 2.75764 1.54688 0.035311 0.0283232 2024-04-22T08:30:39.068 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:39.069 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:39.183 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T08:30:38.217834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588427093,"num_objects":143670,"num_object_clones":0,"num_object_copies":287340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":143670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117272,"num_write_kb":469001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4322,"num_bytes_recovered":17629801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143797,"ondisk_log_size":143797,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1957664,"kb_used_data":1164744,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281157856,"statfs":{"total":289910292480,"available":287905644544,"internally_reserved":0,"allocated":1192697856,"data_stored":1192000508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":811912302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,6,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18472960,"num_objects":4510,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-181,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":741376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008114"},"pg_stats":[{"pgid":"2.f","version":"37'9065","reported_seq":9128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.650852+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:34.650852+0000","last_peered":"2024-04-22T08:30:34.650852+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:34.650852+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37130240,"num_objects":9065,"num_object_clones":0,"num_object_copies":18130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9065,"num_write_kb":36260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5271},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9078","reported_seq":9193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:35.023201+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:35.023201+0000","last_peered":"2024-04-22T08:30:35.023201+0000","last_clean":"2024-04-22T08:30:35.023201+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:35.023201+0000","last_undegraded":"2024-04-22T08:30:35.023201+0000","last_fullsized":"2024-04-22T08:30:35.023201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9078,"log_dups_size":0,"ondisk_log_size":9078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37158912,"num_objects":9072,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":24536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'8913","reported_seq":9036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:35.014844+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:35.014844+0000","last_peered":"2024-04-22T08:30:35.014844+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:35.014844+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36491264,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":22960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9027","reported_seq":9057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.987685+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:34.987685+0000","last_peered":"2024-04-22T08:30:34.987685+0000","last_clean":"2024-04-22T08:30:34.987685+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:34.987685+0000","last_undegraded":"2024-04-22T08:30:34.987685+0000","last_fullsized":"2024-04-22T08:30:34.987685+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36974592,"num_objects":9027,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":36108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'8935","reported_seq":9073,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.694972+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:34.694972+0000","last_peered":"2024-04-22T08:30:34.694972+0000","last_clean":"2024-04-22T08:30:34.694972+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:34.694972+0000","last_undegraded":"2024-04-22T08:30:34.694972+0000","last_fullsized":"2024-04-22T08:30:34.694972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":8935,"log_dups_size":0,"ondisk_log_size":8935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":8928,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6027,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9009","reported_seq":9171,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.646811+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:34.646811+0000","last_peered":"2024-04-22T08:30:34.646811+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:34.646811+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36880384,"num_objects":9004,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":23512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5125},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'8879","reported_seq":16614,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:33.302083+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:33.302083+0000","last_peered":"2024-04-22T08:30:33.302083+0000","last_clean":"2024-04-22T08:30:33.302083+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:33.302083+0000","last_undegraded":"2024-04-22T08:30:33.302083+0000","last_fullsized":"2024-04-22T08:30:33.302083+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36343808,"num_objects":8873,"num_object_clones":0,"num_object_copies":17746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'8899","reported_seq":9043,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:34.630482+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:34.630482+0000","last_peered":"2024-04-22T08:30:34.630482+0000","last_clean":"2024-04-22T08:30:34.630482+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:34.630482+0000","last_undegraded":"2024-04-22T08:30:34.630482+0000","last_fullsized":"2024-04-22T08:30:34.630482+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36442112,"num_objects":8897,"num_object_clones":0,"num_object_copies":17794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":22188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9137","reported_seq":9260,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.959823+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:34.959823+0000","last_peered":"2024-04-22T08:30:34.959823+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:34.959823+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37160146,"num_objects":9074,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":36292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9045","reported_seq":9122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.643879+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:34.643879+0000","last_peered":"2024-04-22T08:30:34.643879+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:34.643879+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016786,"num_objects":9039,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":36152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'8949","reported_seq":9095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.982652+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:34.982652+0000","last_peered":"2024-04-22T08:30:34.982652+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:34.982652+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":8944,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5403,"num_write_kb":21612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'8961","reported_seq":10052,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:35.037023+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:35.037023+0000","last_peered":"2024-04-22T08:30:35.037023+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:35.037023+0000","last_undegraded":"2024-04-22T08:30:35.037023+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680914,"num_objects":8957,"num_object_clones":0,"num_object_copies":17914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":8957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":35823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":521,"num_bytes_recovered":2125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3284},{"shards":"2","objects":5673}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9009","reported_seq":9088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.582990+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:34.582990+0000","last_peered":"2024-04-22T08:30:34.582990+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:34.582990+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36893289,"num_objects":9008,"num_object_clones":0,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":36029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8863","reported_seq":9038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.653162+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:34.653162+0000","last_peered":"2024-04-22T08:30:34.653162+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:34.653162+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8863,"log_dups_size":0,"ondisk_log_size":8863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36270080,"num_objects":8855,"num_object_clones":0,"num_object_copies":17710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":20376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5018},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9007","reported_seq":9083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.632709+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:34.632709+0000","last_peered":"2024-04-22T08:30:34.632709+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:34.632709+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9007,"log_dups_size":0,"ondisk_log_size":9007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36877522,"num_objects":9005,"num_object_clones":0,"num_object_copies":18010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9005,"num_write_kb":36014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5300},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9021","reported_seq":9165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:34.673809+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:34.673809+0000","last_peered":"2024-04-22T08:30:34.673809+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:34.673809+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9021,"log_dups_size":0,"ondisk_log_size":9021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36903332,"num_objects":9013,"num_object_clones":0,"num_object_copies":18026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":36043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5132},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":588427093,"num_objects":143670,"num_object_clones":0,"num_object_copies":287340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3284,"num_objects_unfound":0,"num_objects_dirty":143670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117272,"num_write_kb":469001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4322,"num_bytes_recovered":17629801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1192837120,"data_stored":1192757103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143797,"ondisk_log_size":143797,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117039,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670688,"kb_used_data":408536,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93701152,"statfs":{"total":96636764160,"available":95949979648,"internally_reserved":0,"allocated":418340864,"data_stored":418129454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.437}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":1.4470000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.373}]}]},{"osd":1,"up_from":33,"seq":141733920801,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484596,"kb_used_data":264300,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93887244,"statfs":{"total":96636764160,"available":96140537856,"internally_reserved":0,"allocated":270643200,"data_stored":270362210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":225570508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.254}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":1.78},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.34799999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313267,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802380,"kb_used_data":491908,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93569460,"statfs":{"total":96636764160,"available":95815127040,"internally_reserved":0,"allocated":503713792,"data_stored":503508844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":317910739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.371},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":1.403}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":503779328,"data_stored":503775849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270426112,"data_stored":270353053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418631680,"data_stored":418628201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'9065", 'reported_seq': 9128, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.650852+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:34.650852+0000', 'last_peered': '2024-04-22T08:30:34.650852+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:34.650852+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9065, 'log_dups_size': 0, 'ondisk_log_size': 9065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37130240, 'num_objects': 9065, 'num_object_clones': 0, 'num_object_copies': 18130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9065, 'num_write_kb': 36260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5271}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'8913", 'reported_seq': 9036, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:35.014844+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:35.014844+0000', 'last_peered': '2024-04-22T08:30:35.014844+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:35.014844+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 8913, 'log_dups_size': 0, 'ondisk_log_size': 8913, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36491264, 'num_objects': 8909, 'num_object_clones': 0, 'num_object_copies': 17818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5740, 'num_write_kb': 22960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:39.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9009", 'reported_seq': 9171, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.646811+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:34.646811+0000', 'last_peered': '2024-04-22T08:30:34.646811+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:34.646811+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9009, 'log_dups_size': 0, 'ondisk_log_size': 9009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36880384, 'num_objects': 9004, 'num_object_clones': 0, 'num_object_copies': 18008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5878, 'num_write_kb': 23512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5125}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'9137", 'reported_seq': 9260, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.959823+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:34.959823+0000', 'last_peered': '2024-04-22T08:30:34.959823+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:34.959823+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 9137, 'log_dups_size': 0, 'ondisk_log_size': 9137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37160146, 'num_objects': 9074, 'num_object_clones': 0, 'num_object_copies': 18148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9077, 'num_write_kb': 36292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'9045", 'reported_seq': 9122, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.643879+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:34.643879+0000', 'last_peered': '2024-04-22T08:30:34.643879+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:34.643879+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9045, 'log_dups_size': 0, 'ondisk_log_size': 9045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37016786, 'num_objects': 9039, 'num_object_clones': 0, 'num_object_copies': 18078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9043, 'num_write_kb': 36152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5268}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'8949", 'reported_seq': 9095, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.982652+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:34.982652+0000', 'last_peered': '2024-04-22T08:30:34.982652+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:34.982652+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 8949, 'log_dups_size': 0, 'ondisk_log_size': 8949, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36634624, 'num_objects': 8944, 'num_object_clones': 0, 'num_object_copies': 17888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5403, 'num_write_kb': 21612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5194}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'8961", 'reported_seq': 10052, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:35.037023+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:35.037023+0000', 'last_peered': '2024-04-22T08:30:35.037023+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:35.037023+0000', 'last_undegraded': '2024-04-22T08:30:35.037023+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 8961, 'log_dups_size': 0, 'ondisk_log_size': 8961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36680914, 'num_objects': 8957, 'num_object_clones': 0, 'num_object_copies': 17914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3284, 'num_objects_unfound': 0, 'num_objects_dirty': 8957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8959, 'num_write_kb': 35823, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 521, 'num_bytes_recovered': 2125824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3284}, {'shards': '2', 'objects': 5673}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:39.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'9009", 'reported_seq': 9088, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.582990+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:34.582990+0000', 'last_peered': '2024-04-22T08:30:34.582990+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:34.582990+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9009, 'log_dups_size': 0, 'ondisk_log_size': 9009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36893289, 'num_objects': 9008, 'num_object_clones': 0, 'num_object_copies': 18016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9008, 'num_write_kb': 36029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5287}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'8863", 'reported_seq': 9038, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.653162+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:34.653162+0000', 'last_peered': '2024-04-22T08:30:34.653162+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:34.653162+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8863, 'log_dups_size': 0, 'ondisk_log_size': 8863, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36270080, 'num_objects': 8855, 'num_object_clones': 0, 'num_object_copies': 17710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5094, 'num_write_kb': 20376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5018}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9007", 'reported_seq': 9083, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.632709+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:34.632709+0000', 'last_peered': '2024-04-22T08:30:34.632709+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:34.632709+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9007, 'log_dups_size': 0, 'ondisk_log_size': 9007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36877522, 'num_objects': 9005, 'num_object_clones': 0, 'num_object_copies': 18010, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9005, 'num_write_kb': 36014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5300}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:39.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9021", 'reported_seq': 9165, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:34.673809+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:34.673809+0000', 'last_peered': '2024-04-22T08:30:34.673809+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:34.673809+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9021, 'log_dups_size': 0, 'ondisk_log_size': 9021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36903332, 'num_objects': 9013, 'num_object_clones': 0, 'num_object_copies': 18026, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9018, 'num_write_kb': 36043, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5132}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:39.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:39.887 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 207 20 145940 145920 2.75262 1.71875 0.0256992 0.0283731 2024-04-22T08:30:40.223 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:40.223 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/287340 objects degraded (13.224%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":143670,"data_bytes":588427093,"bytes_used":2004647936,"bytes_avail":287905644544,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":287340,"degraded_ratio":0.13224403146098698,"misplaced_objects":3284,"misplaced_total":287340,"misplaced_ratio":0.0114289691654486,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":61739,"recovering_keys_per_sec":0,"num_objects_recovered":181,"num_bytes_recovered":741376,"num_keys_recovered":0,"write_bytes_sec":1538373,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:30:40.298 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:40.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:40.888 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 208 20 146366 146346 2.74738 1.66406 0.0477221 0.0284257 2024-04-22T08:30:41.158 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:41.164 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:41.214 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-22T08:30:40.218280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597557077,"num_objects":145899,"num_object_clones":0,"num_object_copies":291798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":145899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119501,"num_write_kb":477917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146026,"ondisk_log_size":146026,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1977600,"kb_used_data":1183016,"kb_used_omap":12,"kb_used_meta":794547,"kb_avail":281137920,"statfs":{"total":289910292480,"available":287885230080,"internally_reserved":0,"allocated":1211408384,"data_stored":1210711036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":813616238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,11,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27602944,"num_objects":6739,"num_object_clones":0,"num_object_copies":13478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-287,"num_objects_unfound":0,"num_objects_dirty":6739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":26956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":1175552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008035"},"pg_stats":[{"pgid":"2.f","version":"37'9213","reported_seq":9276,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.757422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:39.757422+0000","last_peered":"2024-04-22T08:30:39.757422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:39.757422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":36852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5419},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9219","reported_seq":9334,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:40.001843+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:40.001843+0000","last_peered":"2024-04-22T08:30:40.001843+0000","last_clean":"2024-04-22T08:30:40.001843+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:40.001843+0000","last_undegraded":"2024-04-22T08:30:40.001843+0000","last_fullsized":"2024-04-22T08:30:40.001843+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9052","reported_seq":9175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:40.050460+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:40.050460+0000","last_peered":"2024-04-22T08:30:40.050460+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:40.050460+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37060608,"num_objects":9048,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9165","reported_seq":9194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.973733+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:39.973733+0000","last_peered":"2024-04-22T08:30:39.973733+0000","last_clean":"2024-04-22T08:30:39.973733+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:39.973733+0000","last_undegraded":"2024-04-22T08:30:39.973733+0000","last_fullsized":"2024-04-22T08:30:39.973733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37539840,"num_objects":9165,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":36660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9098","reported_seq":9236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.745919+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:39.745919+0000","last_peered":"2024-04-22T08:30:39.745919+0000","last_clean":"2024-04-22T08:30:39.745919+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:39.745919+0000","last_undegraded":"2024-04-22T08:30:39.745919+0000","last_fullsized":"2024-04-22T08:30:39.745919+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37236736,"num_objects":9091,"num_object_clones":0,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":24760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9169","reported_seq":9331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.690771+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:39.690771+0000","last_peered":"2024-04-22T08:30:39.690771+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:39.690771+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37535744,"num_objects":9164,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5285},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9020","reported_seq":16756,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:38.344323+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:38.344323+0000","last_peered":"2024-04-22T08:30:38.344323+0000","last_clean":"2024-04-22T08:30:38.344323+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:38.344323+0000","last_undegraded":"2024-04-22T08:30:38.344323+0000","last_fullsized":"2024-04-22T08:30:38.344323+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9020,"log_dups_size":0,"ondisk_log_size":9020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36921344,"num_objects":9014,"num_object_clones":0,"num_object_copies":18028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5388,"num_write_kb":21552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9044","reported_seq":9188,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.685774+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:39.685774+0000","last_peered":"2024-04-22T08:30:39.685774+0000","last_clean":"2024-04-22T08:30:39.685774+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:39.685774+0000","last_undegraded":"2024-04-22T08:30:39.685774+0000","last_fullsized":"2024-04-22T08:30:39.685774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37036032,"num_objects":9042,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9272","reported_seq":9395,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.991725+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:39.991725+0000","last_peered":"2024-04-22T08:30:39.991725+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:39.991725+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713106,"num_objects":9209,"num_object_clones":0,"num_object_copies":18418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":36832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9182","reported_seq":9259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.700139+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:39.700139+0000","last_peered":"2024-04-22T08:30:39.700139+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:39.700139+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37577938,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9180,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5405},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9090","reported_seq":9236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.997587+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:39.997587+0000","last_peered":"2024-04-22T08:30:39.997587+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:39.997587+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37212160,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5544,"num_write_kb":22176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9094","reported_seq":10400,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:40.073289+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:40.073289+0000","last_peered":"2024-04-22T08:30:40.073289+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:40.073289+0000","last_undegraded":"2024-04-22T08:30:40.073289+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37225682,"num_objects":9090,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":9090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":36355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3178},{"shards":"2","objects":5912}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9155","reported_seq":9234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.613599+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:39.613599+0000","last_peered":"2024-04-22T08:30:39.613599+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:39.613599+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491305,"num_objects":9154,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":36613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8986","reported_seq":9161,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.752621+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:39.752621+0000","last_peered":"2024-04-22T08:30:39.752621+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:39.752621+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36773888,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5217,"num_write_kb":20868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5141},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9140","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.705823+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:39.705823+0000","last_peered":"2024-04-22T08:30:39.705823+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:39.705823+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37422290,"num_objects":9138,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":36546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9127","reported_seq":9271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.711714+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:39.711714+0000","last_peered":"2024-04-22T08:30:39.711714+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:39.711714+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37337508,"num_objects":9119,"num_object_clones":0,"num_object_copies":18238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":36467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5238},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":597557077,"num_objects":145899,"num_object_clones":0,"num_object_copies":291798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":145899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119501,"num_write_kb":477917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211559936,"data_stored":1211479919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146026,"ondisk_log_size":146026,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117040,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679344,"kb_used_data":413096,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93692496,"statfs":{"total":96636764160,"available":95941115904,"internally_reserved":0,"allocated":423010304,"data_stored":422798894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":3.8050000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920802,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501476,"kb_used_data":271324,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93870364,"statfs":{"total":96636764160,"available":96123252736,"internally_reserved":0,"allocated":277835776,"data_stored":277554786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":235663052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":4.1980000000000004},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.307}]}]},{"osd":0,"up_from":21,"seq":90194313268,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":796780,"kb_used_data":498596,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93575060,"statfs":{"total":96636764160,"available":95820861440,"internally_reserved":0,"allocated":510562304,"data_stored":510357356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":305327827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":510726144,"data_stored":510722665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277602304,"data_stored":277529245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423231488,"data_stored":423228009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:41.888 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 209 20 146820 146800 2.74273 1.77344 0.0308323 0.0284768 2024-04-22T08:30:42.890 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 210 20 147271 147251 2.73805 1.76172 0.0461336 0.028524 2024-04-22T08:30:43.888 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 211 20 147729 147709 2.73356 1.78906 0.0109546 0.0285707 2024-04-22T08:30:44.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:44.889 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 212 20 148183 148163 2.72901 1.77344 0.0900054 0.0286211 2024-04-22T08:30:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:45.191 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-04-22T08:30:44.221205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598163285,"num_objects":146047,"num_object_clones":0,"num_object_copies":292094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":146047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119649,"num_write_kb":478509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146174,"ondisk_log_size":146174,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1988836,"kb_used_data":1190156,"kb_used_omap":12,"kb_used_meta":798643,"kb_avail":281126684,"statfs":{"total":289910292480,"available":287873724416,"internally_reserved":0,"allocated":1218719744,"data_stored":1218022396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":817810542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,9,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18849792,"num_objects":4602,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-201,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":823296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006439"},"pg_stats":[{"pgid":"2.f","version":"37'9213","reported_seq":9276,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.757422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:39.757422+0000","last_peered":"2024-04-22T08:30:39.757422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:39.757422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":36852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5419},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9219","reported_seq":9334,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:40.001843+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:40.001843+0000","last_peered":"2024-04-22T08:30:40.001843+0000","last_clean":"2024-04-22T08:30:40.001843+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:40.001843+0000","last_undegraded":"2024-04-22T08:30:40.001843+0000","last_fullsized":"2024-04-22T08:30:40.001843+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9052","reported_seq":9175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:40.050460+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:40.050460+0000","last_peered":"2024-04-22T08:30:40.050460+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:40.050460+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37060608,"num_objects":9048,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9165","reported_seq":9194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.973733+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:39.973733+0000","last_peered":"2024-04-22T08:30:39.973733+0000","last_clean":"2024-04-22T08:30:39.973733+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:39.973733+0000","last_undegraded":"2024-04-22T08:30:39.973733+0000","last_fullsized":"2024-04-22T08:30:39.973733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37539840,"num_objects":9165,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":36660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9098","reported_seq":9236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.745919+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:39.745919+0000","last_peered":"2024-04-22T08:30:39.745919+0000","last_clean":"2024-04-22T08:30:39.745919+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:39.745919+0000","last_undegraded":"2024-04-22T08:30:39.745919+0000","last_fullsized":"2024-04-22T08:30:39.745919+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37236736,"num_objects":9091,"num_object_clones":0,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":24760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9169","reported_seq":9331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.690771+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:39.690771+0000","last_peered":"2024-04-22T08:30:39.690771+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:39.690771+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37535744,"num_objects":9164,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5285},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9168","reported_seq":16904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:43.325067+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:43.325067+0000","last_peered":"2024-04-22T08:30:43.325067+0000","last_clean":"2024-04-22T08:30:43.325067+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:43.325067+0000","last_undegraded":"2024-04-22T08:30:43.325067+0000","last_fullsized":"2024-04-22T08:30:43.325067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9168,"log_dups_size":0,"ondisk_log_size":9168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":9162,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9044","reported_seq":9188,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.685774+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:39.685774+0000","last_peered":"2024-04-22T08:30:39.685774+0000","last_clean":"2024-04-22T08:30:39.685774+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:39.685774+0000","last_undegraded":"2024-04-22T08:30:39.685774+0000","last_fullsized":"2024-04-22T08:30:39.685774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37036032,"num_objects":9042,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9272","reported_seq":9395,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.991725+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:39.991725+0000","last_peered":"2024-04-22T08:30:39.991725+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:39.991725+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713106,"num_objects":9209,"num_object_clones":0,"num_object_copies":18418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":36832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9182","reported_seq":9259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.700139+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:39.700139+0000","last_peered":"2024-04-22T08:30:39.700139+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:39.700139+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37577938,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9180,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5405},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9090","reported_seq":9236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.997587+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:39.997587+0000","last_peered":"2024-04-22T08:30:39.997587+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:39.997587+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37212160,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5544,"num_write_kb":22176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9094","reported_seq":10400,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:40.073289+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:40.073289+0000","last_peered":"2024-04-22T08:30:40.073289+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:40.073289+0000","last_undegraded":"2024-04-22T08:30:40.073289+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37225682,"num_objects":9090,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":9090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":36355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3178},{"shards":"2","objects":5912}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9155","reported_seq":9234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.613599+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:39.613599+0000","last_peered":"2024-04-22T08:30:39.613599+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:39.613599+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491305,"num_objects":9154,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":36613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8986","reported_seq":9161,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.752621+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:39.752621+0000","last_peered":"2024-04-22T08:30:39.752621+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:39.752621+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36773888,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5217,"num_write_kb":20868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5141},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9140","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.705823+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:39.705823+0000","last_peered":"2024-04-22T08:30:39.705823+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:39.705823+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37422290,"num_objects":9138,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":36546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9127","reported_seq":9271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.711714+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:39.711714+0000","last_peered":"2024-04-22T08:30:39.711714+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:39.711714+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37337508,"num_objects":9119,"num_object_clones":0,"num_object_copies":18238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":36467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5238},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":598163285,"num_objects":146047,"num_object_clones":0,"num_object_copies":292094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":146047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119649,"num_write_kb":478509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218883584,"data_stored":1218803567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146174,"ondisk_log_size":146174,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117040,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679344,"kb_used_data":413096,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93692496,"statfs":{"total":96636764160,"available":95941115904,"internally_reserved":0,"allocated":423010304,"data_stored":422798894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":3.8050000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920803,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512712,"kb_used_data":278464,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93859128,"statfs":{"total":96636764160,"available":96111747072,"internally_reserved":0,"allocated":285147136,"data_stored":284866146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":239857356},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":5.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313268,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":796780,"kb_used_data":498596,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93575060,"statfs":{"total":96636764160,"available":95820861440,"internally_reserved":0,"allocated":510562304,"data_stored":510357356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":305327827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":510726144,"data_stored":510722665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284925952,"data_stored":284852893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423231488,"data_stored":423228009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:45.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:45.889 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 213 20 148600 148580 2.72385 1.62891 0.0393737 0.0286734 2024-04-22T08:30:46.105 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:46.107 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:46.170 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-04-22T08:30:44.221205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598163285,"num_objects":146047,"num_object_clones":0,"num_object_copies":292094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":146047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119649,"num_write_kb":478509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146174,"ondisk_log_size":146174,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1988836,"kb_used_data":1190156,"kb_used_omap":12,"kb_used_meta":798643,"kb_avail":281126684,"statfs":{"total":289910292480,"available":287873724416,"internally_reserved":0,"allocated":1218719744,"data_stored":1218022396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":817810542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,9,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18849792,"num_objects":4602,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-201,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":823296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006439"},"pg_stats":[{"pgid":"2.f","version":"37'9213","reported_seq":9276,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.757422+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:39.757422+0000","last_peered":"2024-04-22T08:30:39.757422+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:39.757422+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":36852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5419},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9219","reported_seq":9334,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:40.001843+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:40.001843+0000","last_peered":"2024-04-22T08:30:40.001843+0000","last_clean":"2024-04-22T08:30:40.001843+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:40.001843+0000","last_undegraded":"2024-04-22T08:30:40.001843+0000","last_fullsized":"2024-04-22T08:30:40.001843+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736448,"num_objects":9213,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9052","reported_seq":9175,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:40.050460+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:40.050460+0000","last_peered":"2024-04-22T08:30:40.050460+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:40.050460+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37060608,"num_objects":9048,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9165","reported_seq":9194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.973733+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:39.973733+0000","last_peered":"2024-04-22T08:30:39.973733+0000","last_clean":"2024-04-22T08:30:39.973733+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:39.973733+0000","last_undegraded":"2024-04-22T08:30:39.973733+0000","last_fullsized":"2024-04-22T08:30:39.973733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37539840,"num_objects":9165,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":36660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9098","reported_seq":9236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.745919+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:39.745919+0000","last_peered":"2024-04-22T08:30:39.745919+0000","last_clean":"2024-04-22T08:30:39.745919+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:39.745919+0000","last_undegraded":"2024-04-22T08:30:39.745919+0000","last_fullsized":"2024-04-22T08:30:39.745919+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37236736,"num_objects":9091,"num_object_clones":0,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6190,"num_write_kb":24760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9169","reported_seq":9331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.690771+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:39.690771+0000","last_peered":"2024-04-22T08:30:39.690771+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:39.690771+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37535744,"num_objects":9164,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5285},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9168","reported_seq":16904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:43.325067+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:43.325067+0000","last_peered":"2024-04-22T08:30:43.325067+0000","last_clean":"2024-04-22T08:30:43.325067+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:43.325067+0000","last_undegraded":"2024-04-22T08:30:43.325067+0000","last_fullsized":"2024-04-22T08:30:43.325067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9168,"log_dups_size":0,"ondisk_log_size":9168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":9162,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9044","reported_seq":9188,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:39.685774+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:39.685774+0000","last_peered":"2024-04-22T08:30:39.685774+0000","last_clean":"2024-04-22T08:30:39.685774+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:39.685774+0000","last_undegraded":"2024-04-22T08:30:39.685774+0000","last_fullsized":"2024-04-22T08:30:39.685774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37036032,"num_objects":9042,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9272","reported_seq":9395,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.991725+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:39.991725+0000","last_peered":"2024-04-22T08:30:39.991725+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:39.991725+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37713106,"num_objects":9209,"num_object_clones":0,"num_object_copies":18418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":36832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9182","reported_seq":9259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.700139+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:39.700139+0000","last_peered":"2024-04-22T08:30:39.700139+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:39.700139+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37577938,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9180,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5405},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9090","reported_seq":9236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.997587+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:39.997587+0000","last_peered":"2024-04-22T08:30:39.997587+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:39.997587+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37212160,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5544,"num_write_kb":22176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9094","reported_seq":10400,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:40.073289+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:40.073289+0000","last_peered":"2024-04-22T08:30:40.073289+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:40.073289+0000","last_undegraded":"2024-04-22T08:30:40.073289+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9094,"log_dups_size":0,"ondisk_log_size":9094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37225682,"num_objects":9090,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":9090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":36355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3178},{"shards":"2","objects":5912}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9155","reported_seq":9234,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.613599+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:39.613599+0000","last_peered":"2024-04-22T08:30:39.613599+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:39.613599+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491305,"num_objects":9154,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":36613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'8986","reported_seq":9161,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.752621+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:39.752621+0000","last_peered":"2024-04-22T08:30:39.752621+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:39.752621+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36773888,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5217,"num_write_kb":20868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5141},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9140","reported_seq":9216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.705823+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:39.705823+0000","last_peered":"2024-04-22T08:30:39.705823+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:39.705823+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37422290,"num_objects":9138,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":36546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5433},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9127","reported_seq":9271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:39.711714+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:39.711714+0000","last_peered":"2024-04-22T08:30:39.711714+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:39.711714+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37337508,"num_objects":9119,"num_object_clones":0,"num_object_copies":18238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":36467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5238},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":598163285,"num_objects":146047,"num_object_clones":0,"num_object_copies":292094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":146047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119649,"num_write_kb":478509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18063977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218883584,"data_stored":1218803567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146174,"ondisk_log_size":146174,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117040,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679344,"kb_used_data":413096,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93692496,"statfs":{"total":96636764160,"available":95941115904,"internally_reserved":0,"allocated":423010304,"data_stored":422798894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":3.8050000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920803,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512712,"kb_used_data":278464,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93859128,"statfs":{"total":96636764160,"available":96111747072,"internally_reserved":0,"allocated":285147136,"data_stored":284866146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":239857356},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":5.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313268,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":796780,"kb_used_data":498596,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93575060,"statfs":{"total":96636764160,"available":95820861440,"internally_reserved":0,"allocated":510562304,"data_stored":510357356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":305327827},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":510726144,"data_stored":510722665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284925952,"data_stored":284852893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423231488,"data_stored":423228009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:46.171 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:46.171 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'9213", 'reported_seq': 9276, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.757422+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:39.757422+0000', 'last_peered': '2024-04-22T08:30:39.757422+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:39.757422+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9213, 'log_dups_size': 0, 'ondisk_log_size': 9213, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37736448, 'num_objects': 9213, 'num_object_clones': 0, 'num_object_copies': 18426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9213, 'num_write_kb': 36852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5419}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.171 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:46.171 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9052", 'reported_seq': 9175, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:40.050460+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:40.050460+0000', 'last_peered': '2024-04-22T08:30:40.050460+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:40.050460+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9052, 'log_dups_size': 0, 'ondisk_log_size': 9052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37060608, 'num_objects': 9048, 'num_object_clones': 0, 'num_object_copies': 18096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5879, 'num_write_kb': 23516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9169", 'reported_seq': 9331, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.690771+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:39.690771+0000', 'last_peered': '2024-04-22T08:30:39.690771+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:39.690771+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9169, 'log_dups_size': 0, 'ondisk_log_size': 9169, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37535744, 'num_objects': 9164, 'num_object_clones': 0, 'num_object_copies': 18328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6038, 'num_write_kb': 24152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5285}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'9272", 'reported_seq': 9395, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.991725+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:39.991725+0000', 'last_peered': '2024-04-22T08:30:39.991725+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:39.991725+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 9272, 'log_dups_size': 0, 'ondisk_log_size': 9272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37713106, 'num_objects': 9209, 'num_object_clones': 0, 'num_object_copies': 18418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9212, 'num_write_kb': 36832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'9182", 'reported_seq': 9259, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.700139+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:39.700139+0000', 'last_peered': '2024-04-22T08:30:39.700139+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:39.700139+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9182, 'log_dups_size': 0, 'ondisk_log_size': 9182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37577938, 'num_objects': 9176, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9180, 'num_write_kb': 36700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5405}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'9090", 'reported_seq': 9236, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.997587+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:39.997587+0000', 'last_peered': '2024-04-22T08:30:39.997587+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:39.997587+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 9090, 'log_dups_size': 0, 'ondisk_log_size': 9090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37212160, 'num_objects': 9085, 'num_object_clones': 0, 'num_object_copies': 18170, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5544, 'num_write_kb': 22176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5335}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:46.172 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'9094", 'reported_seq': 10400, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:40.073289+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:40.073289+0000', 'last_peered': '2024-04-22T08:30:40.073289+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:40.073289+0000', 'last_undegraded': '2024-04-22T08:30:40.073289+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9094, 'log_dups_size': 0, 'ondisk_log_size': 9094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37225682, 'num_objects': 9090, 'num_object_clones': 0, 'num_object_copies': 18180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3178, 'num_objects_unfound': 0, 'num_objects_dirty': 9090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9092, 'num_write_kb': 36355, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 627, 'num_bytes_recovered': 2560000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3178}, {'shards': '2', 'objects': 5912}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'9155", 'reported_seq': 9234, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.613599+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:39.613599+0000', 'last_peered': '2024-04-22T08:30:39.613599+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:39.613599+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9155, 'log_dups_size': 0, 'ondisk_log_size': 9155, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37491305, 'num_objects': 9154, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9154, 'num_write_kb': 36613, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5433}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'8986", 'reported_seq': 9161, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.752621+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:39.752621+0000', 'last_peered': '2024-04-22T08:30:39.752621+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:39.752621+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 8986, 'log_dups_size': 0, 'ondisk_log_size': 8986, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36773888, 'num_objects': 8978, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5217, 'num_write_kb': 20868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5141}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9140", 'reported_seq': 9216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.705823+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:39.705823+0000', 'last_peered': '2024-04-22T08:30:39.705823+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:39.705823+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9140, 'log_dups_size': 0, 'ondisk_log_size': 9140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37422290, 'num_objects': 9138, 'num_object_clones': 0, 'num_object_copies': 18276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9138, 'num_write_kb': 36546, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5433}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:46.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9127", 'reported_seq': 9271, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:39.711714+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:39.711714+0000', 'last_peered': '2024-04-22T08:30:39.711714+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:39.711714+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9127, 'log_dups_size': 0, 'ondisk_log_size': 9127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37337508, 'num_objects': 9119, 'num_object_clones': 0, 'num_object_copies': 18238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9124, 'num_write_kb': 36467, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5238}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:46.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:46.889 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 214 20 149021 149001 2.71881 1.64453 0.0295016 0.0287267 2024-04-22T08:30:47.260 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:47.263 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/296322 objects degraded (12.824%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":282,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":148161,"data_bytes":606822229,"bytes_used":2060419072,"bytes_avail":287849873408,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":296322,"degraded_ratio":0.12823550057032551,"misplaced_objects":3073,"misplaced_total":296322,"misplaced_ratio":0.010370475361262409,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":104392,"recovering_keys_per_sec":0,"num_objects_recovered":306,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":2248527,"read_op_per_sec":0,"write_op_per_sec":548},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:30:47.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:47.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:47.890 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 215 20 149437 149417 2.71372 1.625 0.100899 0.0287787 2024-04-22T08:30:48.145 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:48.197 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-04-22T08:30:46.221702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606822229,"num_objects":148161,"num_object_clones":0,"num_object_copies":296322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":148161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121763,"num_write_kb":486965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18494057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148288,"ondisk_log_size":148288,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2012128,"kb_used_data":1201160,"kb_used_omap":12,"kb_used_meta":810931,"kb_avail":281103392,"statfs":{"total":289910292480,"available":287849873408,"internally_reserved":0,"allocated":1229987840,"data_stored":1229290492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":830393454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,8,1,9,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26996736,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-306,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":26364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006408"},"pg_stats":[{"pgid":"2.f","version":"37'9344","reported_seq":9407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.743256+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:44.743256+0000","last_peered":"2024-04-22T08:30:44.743256+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:44.743256+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273024,"num_objects":9344,"num_object_clones":0,"num_object_copies":18688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":37376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5550},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9371","reported_seq":9484,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:45.046442+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:45.046442+0000","last_peered":"2024-04-22T08:30:45.046442+0000","last_clean":"2024-04-22T08:30:45.046442+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:45.046442+0000","last_undegraded":"2024-04-22T08:30:45.046442+0000","last_fullsized":"2024-04-22T08:30:45.046442+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9371,"log_dups_size":0,"ondisk_log_size":9371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38359040,"num_objects":9365,"num_object_clones":0,"num_object_copies":18730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":25708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9205","reported_seq":9325,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.143560+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:45.143560+0000","last_peered":"2024-04-22T08:30:45.143560+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:45.143560+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37687296,"num_objects":9201,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6032,"num_write_kb":24128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9312","reported_seq":9342,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:45.133666+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:45.133666+0000","last_peered":"2024-04-22T08:30:45.133666+0000","last_clean":"2024-04-22T08:30:45.133666+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:45.133666+0000","last_undegraded":"2024-04-22T08:30:45.133666+0000","last_fullsized":"2024-04-22T08:30:45.133666+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":9312,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9231","reported_seq":9369,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:44.787144+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:44.787144+0000","last_peered":"2024-04-22T08:30:44.787144+0000","last_clean":"2024-04-22T08:30:44.787144+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:44.787144+0000","last_undegraded":"2024-04-22T08:30:44.787144+0000","last_fullsized":"2024-04-22T08:30:44.787144+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37781504,"num_objects":9224,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6323,"num_write_kb":25292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9293","reported_seq":9455,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.681631+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:44.681631+0000","last_peered":"2024-04-22T08:30:44.681631+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:44.681631+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9293,"log_dups_size":0,"ondisk_log_size":9293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38043648,"num_objects":9288,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5409},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9168","reported_seq":16904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:43.325067+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:43.325067+0000","last_peered":"2024-04-22T08:30:43.325067+0000","last_clean":"2024-04-22T08:30:43.325067+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:43.325067+0000","last_undegraded":"2024-04-22T08:30:43.325067+0000","last_fullsized":"2024-04-22T08:30:43.325067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9168,"log_dups_size":0,"ondisk_log_size":9168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":9162,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9187","reported_seq":9330,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:44.793817+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:44.793817+0000","last_peered":"2024-04-22T08:30:44.793817+0000","last_clean":"2024-04-22T08:30:44.793817+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:44.793817+0000","last_undegraded":"2024-04-22T08:30:44.793817+0000","last_fullsized":"2024-04-22T08:30:44.793817+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37621760,"num_objects":9185,"num_object_clones":0,"num_object_copies":18370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":23340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9409","reported_seq":9532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.075254+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:45.075254+0000","last_peered":"2024-04-22T08:30:45.075254+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:45.075254+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9409,"log_dups_size":0,"ondisk_log_size":9409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38274258,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":37380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9327","reported_seq":9404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.698722+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:44.698722+0000","last_peered":"2024-04-22T08:30:44.698722+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:44.698722+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38171858,"num_objects":9321,"num_object_clones":0,"num_object_copies":18642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":37280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5550},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9235","reported_seq":9381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.115601+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:45.115601+0000","last_peered":"2024-04-22T08:30:45.115601+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:45.115601+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37806080,"num_objects":9230,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5689,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9226","reported_seq":10741,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:45.137716+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:45.137716+0000","last_peered":"2024-04-22T08:30:45.137716+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:45.137716+0000","last_undegraded":"2024-04-22T08:30:45.137716+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37766354,"num_objects":9222,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":9222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":36883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":2990080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3073},{"shards":"2","objects":6149}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9293","reported_seq":9372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.788929+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:44.788929+0000","last_peered":"2024-04-22T08:30:44.788929+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:44.788929+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9293,"log_dups_size":0,"ondisk_log_size":9293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38056553,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":37165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5571},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9122","reported_seq":9297,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.776396+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:44.776396+0000","last_peered":"2024-04-22T08:30:44.776396+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:44.776396+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9122,"log_dups_size":0,"ondisk_log_size":9122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37330944,"num_objects":9114,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":21412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5277},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9279","reported_seq":9355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.789422+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:44.789422+0000","last_peered":"2024-04-22T08:30:44.789422+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:44.789422+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37991634,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":37102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5572},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9286","reported_seq":9430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:44.699123+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:44.699123+0000","last_peered":"2024-04-22T08:30:44.699123+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:44.699123+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9286,"log_dups_size":0,"ondisk_log_size":9286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37988772,"num_objects":9278,"num_object_clones":0,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9283,"num_write_kb":37103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5397},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":606822229,"num_objects":148161,"num_object_clones":0,"num_object_copies":296322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":148161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121763,"num_write_kb":486965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18494057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230475264,"data_stored":1230395247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148288,"ondisk_log_size":148288,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117041,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687888,"kb_used_data":417544,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93683952,"statfs":{"total":96636764160,"available":95932366848,"internally_reserved":0,"allocated":427565056,"data_stored":427353646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.48799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920803,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512712,"kb_used_data":278464,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93859128,"statfs":{"total":96636764160,"available":96111747072,"internally_reserved":0,"allocated":285147136,"data_stored":284866146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":239857356},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":6.0999999999999996},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":5.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313269,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811528,"kb_used_data":505152,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93560312,"statfs":{"total":96636764160,"available":95805759488,"internally_reserved":0,"allocated":517275648,"data_stored":517070700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":313716435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,6,0,6,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":4.4450000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":517611520,"data_stored":517608041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284925952,"data_stored":284852893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427937792,"data_stored":427934313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:48.893 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 216 20 149885 149865 2.70922 1.75 0.0739083 0.0288293 2024-04-22T08:30:49.893 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 217 19 150338 150319 2.7049 1.77344 0.0611748 0.028875 2024-04-22T08:30:50.893 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 218 20 150778 150758 2.70036 1.71484 0.0765442 0.0289241 2024-04-22T08:30:51.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:51.893 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 219 20 151203 151183 2.69561 1.66016 0.0574744 0.0289751 2024-04-22T08:30:52.113 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:52.119 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:52.178 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-04-22T08:30:50.222686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612331349,"num_objects":149506,"num_object_clones":0,"num_object_copies":299012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":149506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123108,"num_write_kb":492345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18494057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149633,"ondisk_log_size":149633,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2039336,"kb_used_data":1214416,"kb_used_omap":12,"kb_used_meta":824883,"kb_avail":281076184,"statfs":{"total":289910292480,"available":287822012416,"internally_reserved":0,"allocated":1243561984,"data_stored":1242864636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":844680302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,5,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23904256,"num_objects":5836,"num_object_clones":0,"num_object_copies":11672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-211,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5836,"num_write_kb":23344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":864256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004852"},"pg_stats":[{"pgid":"2.f","version":"37'9488","reported_seq":9551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.778223+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:49.778223+0000","last_peered":"2024-04-22T08:30:49.778223+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:49.778223+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":9488,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":37952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5694},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9371","reported_seq":9484,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:45.046442+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:45.046442+0000","last_peered":"2024-04-22T08:30:45.046442+0000","last_clean":"2024-04-22T08:30:45.046442+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:45.046442+0000","last_undegraded":"2024-04-22T08:30:45.046442+0000","last_fullsized":"2024-04-22T08:30:45.046442+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9371,"log_dups_size":0,"ondisk_log_size":9371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38359040,"num_objects":9365,"num_object_clones":0,"num_object_copies":18730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":25708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9205","reported_seq":9325,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.143560+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:45.143560+0000","last_peered":"2024-04-22T08:30:45.143560+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:45.143560+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37687296,"num_objects":9201,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6032,"num_write_kb":24128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9312","reported_seq":9342,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:45.133666+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:45.133666+0000","last_peered":"2024-04-22T08:30:45.133666+0000","last_clean":"2024-04-22T08:30:45.133666+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:45.133666+0000","last_undegraded":"2024-04-22T08:30:45.133666+0000","last_fullsized":"2024-04-22T08:30:45.133666+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":9312,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9381","reported_seq":9519,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.789762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:49.789762+0000","last_peered":"2024-04-22T08:30:49.789762+0000","last_clean":"2024-04-22T08:30:49.789762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:49.789762+0000","last_undegraded":"2024-04-22T08:30:49.789762+0000","last_fullsized":"2024-04-22T08:30:49.789762+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38395904,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9438","reported_seq":9600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.810686+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:49.810686+0000","last_peered":"2024-04-22T08:30:49.810686+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:49.810686+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38637568,"num_objects":9433,"num_object_clones":0,"num_object_copies":18866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5554},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9290","reported_seq":17026,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:48.439835+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:48.439835+0000","last_peered":"2024-04-22T08:30:48.439835+0000","last_clean":"2024-04-22T08:30:48.439835+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:48.439835+0000","last_undegraded":"2024-04-22T08:30:48.439835+0000","last_fullsized":"2024-04-22T08:30:48.439835+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38027264,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":22632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9311","reported_seq":9455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.731331+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:49.731331+0000","last_peered":"2024-04-22T08:30:49.731331+0000","last_clean":"2024-04-22T08:30:49.731331+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:49.731331+0000","last_undegraded":"2024-04-22T08:30:49.731331+0000","last_fullsized":"2024-04-22T08:30:49.731331+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9311,"log_dups_size":0,"ondisk_log_size":9311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5959,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9409","reported_seq":9532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.075254+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:45.075254+0000","last_peered":"2024-04-22T08:30:45.075254+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:45.075254+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9409,"log_dups_size":0,"ondisk_log_size":9409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38274258,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":37380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9465","reported_seq":9542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.751614+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:49.751614+0000","last_peered":"2024-04-22T08:30:49.751614+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:49.751614+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38737106,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9463,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5688},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9235","reported_seq":9381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:45.115601+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:45.115601+0000","last_peered":"2024-04-22T08:30:45.115601+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:45.115601+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37806080,"num_objects":9230,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5689,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9226","reported_seq":10741,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:45.137716+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:45.137716+0000","last_peered":"2024-04-22T08:30:45.137716+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:45.137716+0000","last_undegraded":"2024-04-22T08:30:45.137716+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37766354,"num_objects":9222,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":9222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":36883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":2990080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3073},{"shards":"2","objects":6149}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9432","reported_seq":9511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.866497+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:49.866497+0000","last_peered":"2024-04-22T08:30:49.866497+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:49.866497+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38625897,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":37721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9249","reported_seq":9424,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.626877+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:49.626877+0000","last_peered":"2024-04-22T08:30:49.626877+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:49.626877+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37851136,"num_objects":9241,"num_object_clones":0,"num_object_copies":18482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5480,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5404},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9405","reported_seq":9479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.834160+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:49.834160+0000","last_peered":"2024-04-22T08:30:49.834160+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:49.834160+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38507730,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5698},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9416","reported_seq":9560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.875332+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:49.875332+0000","last_peered":"2024-04-22T08:30:49.875332+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:49.875332+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521252,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":37623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5527},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":612331349,"num_objects":149506,"num_object_clones":0,"num_object_copies":299012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":3073,"num_objects_unfound":0,"num_objects_dirty":149506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123108,"num_write_kb":492345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18494057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1244229632,"data_stored":1244149615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149633,"ondisk_log_size":149633,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117041,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687888,"kb_used_data":417544,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93683952,"statfs":{"total":96636764160,"available":95932366848,"internally_reserved":0,"allocated":427565056,"data_stored":427353646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.48799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920804,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529468,"kb_used_data":285364,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93842372,"statfs":{"total":96636764160,"available":96094588928,"internally_reserved":0,"allocated":292212736,"data_stored":291931746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":249949900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.314}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.372}]}]},{"osd":0,"up_from":21,"seq":90194313270,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821980,"kb_used_data":511508,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93549860,"statfs":{"total":96636764160,"available":95795056640,"internally_reserved":0,"allocated":523784192,"data_stored":523579244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":317910739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.157},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":6.8730000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524316672,"data_stored":524313193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291975168,"data_stored":291902109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427937792,"data_stored":427934313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:52.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:52.894 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:2024-04-22T08:30:52.896017+0000 min lat: 0.00109492 max lat: 2.74727 avg lat: 0.029023 2024-04-22T08:30:52.894 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T08:30:52.894 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 220 20 151638 151618 2.69108 1.69922 0.0717851 0.029023 2024-04-22T08:30:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:52.957 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:53.004 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-04-22T08:30:52.228629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615751509,"num_objects":150341,"num_object_clones":0,"num_object_copies":300682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":150341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123943,"num_write_kb":495685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18952809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150468,"ondisk_log_size":150468,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2047864,"kb_used_data":1218848,"kb_used_omap":12,"kb_used_meta":828979,"kb_avail":281067656,"statfs":{"total":289910292480,"available":287813279744,"internally_reserved":0,"allocated":1248100352,"data_stored":1247403004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":848874606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,7,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18194432,"num_objects":4442,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-217,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":17768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010348"},"pg_stats":[{"pgid":"2.f","version":"37'9488","reported_seq":9551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.778223+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:49.778223+0000","last_peered":"2024-04-22T08:30:49.778223+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:49.778223+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":9488,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":37952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5694},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9502","reported_seq":9617,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:50.059447+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:50.059447+0000","last_peered":"2024-04-22T08:30:50.059447+0000","last_clean":"2024-04-22T08:30:50.059447+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:50.059447+0000","last_undegraded":"2024-04-22T08:30:50.059447+0000","last_fullsized":"2024-04-22T08:30:50.059447+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38895616,"num_objects":9496,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":26232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9345","reported_seq":9468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.151942+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:50.151942+0000","last_peered":"2024-04-22T08:30:50.151942+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:50.151942+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38260736,"num_objects":9341,"num_object_clones":0,"num_object_copies":18682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":24688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9458","reported_seq":9488,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:50.184855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:50.184855+0000","last_peered":"2024-04-22T08:30:50.184855+0000","last_clean":"2024-04-22T08:30:50.184855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:50.184855+0000","last_undegraded":"2024-04-22T08:30:50.184855+0000","last_fullsized":"2024-04-22T08:30:50.184855+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38739968,"num_objects":9458,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9381","reported_seq":9519,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.789762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:49.789762+0000","last_peered":"2024-04-22T08:30:49.789762+0000","last_clean":"2024-04-22T08:30:49.789762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:49.789762+0000","last_undegraded":"2024-04-22T08:30:49.789762+0000","last_fullsized":"2024-04-22T08:30:49.789762+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38395904,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9438","reported_seq":9600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.810686+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:49.810686+0000","last_peered":"2024-04-22T08:30:49.810686+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:49.810686+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38637568,"num_objects":9433,"num_object_clones":0,"num_object_copies":18866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5554},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9290","reported_seq":17026,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:48.439835+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:48.439835+0000","last_peered":"2024-04-22T08:30:48.439835+0000","last_clean":"2024-04-22T08:30:48.439835+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:48.439835+0000","last_undegraded":"2024-04-22T08:30:48.439835+0000","last_fullsized":"2024-04-22T08:30:48.439835+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38027264,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":22632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9311","reported_seq":9455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.731331+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:49.731331+0000","last_peered":"2024-04-22T08:30:49.731331+0000","last_clean":"2024-04-22T08:30:49.731331+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:49.731331+0000","last_undegraded":"2024-04-22T08:30:49.731331+0000","last_fullsized":"2024-04-22T08:30:49.731331+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9311,"log_dups_size":0,"ondisk_log_size":9311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5959,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9559","reported_seq":9682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.202561+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:50.202561+0000","last_peered":"2024-04-22T08:30:50.202561+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:50.202561+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38888658,"num_objects":9496,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":37980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5608}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9465","reported_seq":9542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.751614+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:49.751614+0000","last_peered":"2024-04-22T08:30:49.751614+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:49.751614+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38737106,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9463,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5688},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9369","reported_seq":9515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.103040+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:50.103040+0000","last_peered":"2024-04-22T08:30:50.103040+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:50.103040+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38354944,"num_objects":9364,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":23292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5614}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9360","reported_seq":11100,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:50.138693+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:50.138693+0000","last_peered":"2024-04-22T08:30:50.138693+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:50.138693+0000","last_undegraded":"2024-04-22T08:30:50.138693+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38315218,"num_objects":9356,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":9356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9358,"num_write_kb":37419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2961},{"shards":"2","objects":6395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9432","reported_seq":9511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.866497+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:49.866497+0000","last_peered":"2024-04-22T08:30:49.866497+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:49.866497+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38625897,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":37721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9249","reported_seq":9424,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.626877+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:49.626877+0000","last_peered":"2024-04-22T08:30:49.626877+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:49.626877+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37851136,"num_objects":9241,"num_object_clones":0,"num_object_copies":18482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5480,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5404},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9405","reported_seq":9479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.834160+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:49.834160+0000","last_peered":"2024-04-22T08:30:49.834160+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:49.834160+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38507730,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5698},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9416","reported_seq":9560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.875332+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:49.875332+0000","last_peered":"2024-04-22T08:30:49.875332+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:49.875332+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521252,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":37623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5527},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":615751509,"num_objects":150341,"num_object_clones":0,"num_object_copies":300682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":150341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123943,"num_write_kb":495685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18952809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1248759808,"data_stored":1248679791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150468,"ondisk_log_size":150468,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696416,"kb_used_data":421976,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93675424,"statfs":{"total":96636764160,"available":95923634176,"internally_reserved":0,"allocated":432103424,"data_stored":431892014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":3.8210000000000002},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":5.4029999999999996}]}]},{"osd":1,"up_from":33,"seq":141733920804,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529468,"kb_used_data":285364,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93842372,"statfs":{"total":96636764160,"available":96094588928,"internally_reserved":0,"allocated":292212736,"data_stored":291931746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":249949900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.314}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.372}]}]},{"osd":0,"up_from":21,"seq":90194313270,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821980,"kb_used_data":511508,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93549860,"statfs":{"total":96636764160,"available":95795056640,"internally_reserved":0,"allocated":523784192,"data_stored":523579244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":317910739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.157},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":6.8730000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524316672,"data_stored":524313193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291975168,"data_stored":291902109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432467968,"data_stored":432464489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'9488", 'reported_seq': 9551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.778223+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:49.778223+0000', 'last_peered': '2024-04-22T08:30:49.778223+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:49.778223+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9488, 'log_dups_size': 0, 'ondisk_log_size': 9488, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38862848, 'num_objects': 9488, 'num_object_clones': 0, 'num_object_copies': 18976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9488, 'num_write_kb': 37952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5694}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9345", 'reported_seq': 9468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:50.151942+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:50.151942+0000', 'last_peered': '2024-04-22T08:30:50.151942+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:50.151942+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9345, 'log_dups_size': 0, 'ondisk_log_size': 9345, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38260736, 'num_objects': 9341, 'num_object_clones': 0, 'num_object_copies': 18682, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6172, 'num_write_kb': 24688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:53.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9438", 'reported_seq': 9600, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.810686+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:49.810686+0000', 'last_peered': '2024-04-22T08:30:49.810686+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:49.810686+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9438, 'log_dups_size': 0, 'ondisk_log_size': 9438, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38637568, 'num_objects': 9433, 'num_object_clones': 0, 'num_object_copies': 18866, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6307, 'num_write_kb': 25228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5554}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'9559", 'reported_seq': 9682, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:50.202561+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:50.202561+0000', 'last_peered': '2024-04-22T08:30:50.202561+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:50.202561+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 9559, 'log_dups_size': 0, 'ondisk_log_size': 9559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38888658, 'num_objects': 9496, 'num_object_clones': 0, 'num_object_copies': 18992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9499, 'num_write_kb': 37980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'9465", 'reported_seq': 9542, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.751614+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:49.751614+0000', 'last_peered': '2024-04-22T08:30:49.751614+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:49.751614+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9465, 'log_dups_size': 0, 'ondisk_log_size': 9465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38737106, 'num_objects': 9459, 'num_object_clones': 0, 'num_object_copies': 18918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9463, 'num_write_kb': 37832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5688}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'9369", 'reported_seq': 9515, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:50.103040+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:50.103040+0000', 'last_peered': '2024-04-22T08:30:50.103040+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:50.103040+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 9369, 'log_dups_size': 0, 'ondisk_log_size': 9369, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38354944, 'num_objects': 9364, 'num_object_clones': 0, 'num_object_copies': 18728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9364, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5823, 'num_write_kb': 23292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5614}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'9360", 'reported_seq': 11100, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:50.138693+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:50.138693+0000', 'last_peered': '2024-04-22T08:30:50.138693+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:50.138693+0000', 'last_undegraded': '2024-04-22T08:30:50.138693+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9360, 'log_dups_size': 0, 'ondisk_log_size': 9360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38315218, 'num_objects': 9356, 'num_object_clones': 0, 'num_object_copies': 18712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2961, 'num_objects_unfound': 0, 'num_objects_dirty': 9356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9358, 'num_write_kb': 37419, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 844, 'num_bytes_recovered': 3448832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2961}, {'shards': '2', 'objects': 6395}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:53.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'9432", 'reported_seq': 9511, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.866497+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:49.866497+0000', 'last_peered': '2024-04-22T08:30:49.866497+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:49.866497+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9432, 'log_dups_size': 0, 'ondisk_log_size': 9432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38625897, 'num_objects': 9431, 'num_object_clones': 0, 'num_object_copies': 18862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9431, 'num_write_kb': 37721, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5710}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'9249", 'reported_seq': 9424, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.626877+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:49.626877+0000', 'last_peered': '2024-04-22T08:30:49.626877+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:49.626877+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 9249, 'log_dups_size': 0, 'ondisk_log_size': 9249, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37851136, 'num_objects': 9241, 'num_object_clones': 0, 'num_object_copies': 18482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5480, 'num_write_kb': 21920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5404}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9405", 'reported_seq': 9479, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.834160+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:49.834160+0000', 'last_peered': '2024-04-22T08:30:49.834160+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:49.834160+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9405, 'log_dups_size': 0, 'ondisk_log_size': 9405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38507730, 'num_objects': 9403, 'num_object_clones': 0, 'num_object_copies': 18806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9403, 'num_write_kb': 37606, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5698}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:53.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9416", 'reported_seq': 9560, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:49.875332+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:49.875332+0000', 'last_peered': '2024-04-22T08:30:49.875332+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:49.875332+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9416, 'log_dups_size': 0, 'ondisk_log_size': 9416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38521252, 'num_objects': 9408, 'num_object_clones': 0, 'num_object_copies': 18816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9413, 'num_write_kb': 37623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5527}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:53.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:53.895 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 221 20 152063 152043 2.6864 1.66016 0.0171341 0.0290736 2024-04-22T08:30:54.144 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:54.144 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/300682 objects degraded (12.638%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":289,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":150341,"data_bytes":615751509,"bytes_used":2097012736,"bytes_avail":287813279744,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":300682,"degraded_ratio":0.12637603847253909,"misplaced_objects":2961,"misplaced_total":300682,"misplaced_ratio":0.0098476130929021355,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":74005,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"write_bytes_sec":1514896,"read_op_per_sec":0,"write_op_per_sec":369},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:30:54.180 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:30:54.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:54.895 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 222 20 152495 152475 2.6819 1.6875 0.0620481 0.0291248 2024-04-22T08:30:55.058 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:55.058 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:55.107 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-04-22T08:30:54.229171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616419157,"num_objects":150504,"num_object_clones":0,"num_object_copies":301008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":150504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124106,"num_write_kb":496337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18952809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150631,"ondisk_log_size":150631,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2059140,"kb_used_data":1226028,"kb_used_omap":12,"kb_used_meta":833075,"kb_avail":281056380,"statfs":{"total":289910292480,"available":287801733120,"internally_reserved":0,"allocated":1255452672,"data_stored":1254755324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":853068910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,8,11,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18862080,"num_objects":4605,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-217,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4605,"num_write_kb":18420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010369"},"pg_stats":[{"pgid":"2.f","version":"37'9488","reported_seq":9551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.778223+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:49.778223+0000","last_peered":"2024-04-22T08:30:49.778223+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:49.778223+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":9488,"num_object_clones":0,"num_object_copies":18976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":37952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5694},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9502","reported_seq":9617,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:50.059447+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:50.059447+0000","last_peered":"2024-04-22T08:30:50.059447+0000","last_clean":"2024-04-22T08:30:50.059447+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:50.059447+0000","last_undegraded":"2024-04-22T08:30:50.059447+0000","last_fullsized":"2024-04-22T08:30:50.059447+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38895616,"num_objects":9496,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":26232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9345","reported_seq":9468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.151942+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:50.151942+0000","last_peered":"2024-04-22T08:30:50.151942+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:50.151942+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38260736,"num_objects":9341,"num_object_clones":0,"num_object_copies":18682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":24688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9458","reported_seq":9488,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:50.184855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:50.184855+0000","last_peered":"2024-04-22T08:30:50.184855+0000","last_clean":"2024-04-22T08:30:50.184855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:50.184855+0000","last_undegraded":"2024-04-22T08:30:50.184855+0000","last_fullsized":"2024-04-22T08:30:50.184855+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38739968,"num_objects":9458,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9381","reported_seq":9519,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.789762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:49.789762+0000","last_peered":"2024-04-22T08:30:49.789762+0000","last_clean":"2024-04-22T08:30:49.789762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:49.789762+0000","last_undegraded":"2024-04-22T08:30:49.789762+0000","last_fullsized":"2024-04-22T08:30:49.789762+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38395904,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9438","reported_seq":9600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.810686+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:49.810686+0000","last_peered":"2024-04-22T08:30:49.810686+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:49.810686+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38637568,"num_objects":9433,"num_object_clones":0,"num_object_copies":18866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":25228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5554},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9453","reported_seq":17189,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:53.548573+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:53.548573+0000","last_peered":"2024-04-22T08:30:53.548573+0000","last_clean":"2024-04-22T08:30:53.548573+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:53.548573+0000","last_undegraded":"2024-04-22T08:30:53.548573+0000","last_fullsized":"2024-04-22T08:30:53.548573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38694912,"num_objects":9447,"num_object_clones":0,"num_object_copies":18894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":23284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9311","reported_seq":9455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:49.731331+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:49.731331+0000","last_peered":"2024-04-22T08:30:49.731331+0000","last_clean":"2024-04-22T08:30:49.731331+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:49.731331+0000","last_undegraded":"2024-04-22T08:30:49.731331+0000","last_fullsized":"2024-04-22T08:30:49.731331+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9311,"log_dups_size":0,"ondisk_log_size":9311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5959,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9559","reported_seq":9682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.202561+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:50.202561+0000","last_peered":"2024-04-22T08:30:50.202561+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:50.202561+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38888658,"num_objects":9496,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":37980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5608}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9465","reported_seq":9542,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.751614+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:49.751614+0000","last_peered":"2024-04-22T08:30:49.751614+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:49.751614+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9465,"log_dups_size":0,"ondisk_log_size":9465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38737106,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9463,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5688},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9369","reported_seq":9515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:50.103040+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:50.103040+0000","last_peered":"2024-04-22T08:30:50.103040+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:50.103040+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38354944,"num_objects":9364,"num_object_clones":0,"num_object_copies":18728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":23292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5614}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9360","reported_seq":11100,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:50.138693+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:50.138693+0000","last_peered":"2024-04-22T08:30:50.138693+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:50.138693+0000","last_undegraded":"2024-04-22T08:30:50.138693+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38315218,"num_objects":9356,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":9356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9358,"num_write_kb":37419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":3448832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2961},{"shards":"2","objects":6395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9432","reported_seq":9511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.866497+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:49.866497+0000","last_peered":"2024-04-22T08:30:49.866497+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:49.866497+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38625897,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":37721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9249","reported_seq":9424,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.626877+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:49.626877+0000","last_peered":"2024-04-22T08:30:49.626877+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:49.626877+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37851136,"num_objects":9241,"num_object_clones":0,"num_object_copies":18482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5480,"num_write_kb":21920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5404},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9405","reported_seq":9479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.834160+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:49.834160+0000","last_peered":"2024-04-22T08:30:49.834160+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:49.834160+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38507730,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5698},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9416","reported_seq":9560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:49.875332+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:49.875332+0000","last_peered":"2024-04-22T08:30:49.875332+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:49.875332+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38521252,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":37623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5527},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":616419157,"num_objects":150504,"num_object_clones":0,"num_object_copies":301008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2961,"num_objects_unfound":0,"num_objects_dirty":150504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124106,"num_write_kb":496337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18952809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1256042496,"data_stored":1255962479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150631,"ondisk_log_size":150631,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117042,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696416,"kb_used_data":421976,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93675424,"statfs":{"total":96636764160,"available":95923634176,"internally_reserved":0,"allocated":432103424,"data_stored":431892014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":3.8210000000000002},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":5.4029999999999996}]}]},{"osd":1,"up_from":33,"seq":141733920805,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540744,"kb_used_data":292544,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93831096,"statfs":{"total":96636764160,"available":96083042304,"internally_reserved":0,"allocated":299565056,"data_stored":299284066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":254144204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.1079999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":15.989000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":20.149000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313270,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821980,"kb_used_data":511508,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93549860,"statfs":{"total":96636764160,"available":95795056640,"internally_reserved":0,"allocated":523784192,"data_stored":523579244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":317910739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":1.157},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":6.8730000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524316672,"data_stored":524313193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299257856,"data_stored":299184797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432467968,"data_stored":432464489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:55.904 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 223 20 152925 152905 2.67735 1.67969 0.0357801 0.0291731 2024-04-22T08:30:56.903 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 224 20 153380 153360 2.67329 1.77734 0.0273094 0.0292181 2024-04-22T08:30:57.903 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 225 20 153824 153804 2.66912 1.73438 0.0550572 0.0292636 2024-04-22T08:30:58.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:58.882 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:58.904 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 226 20 154244 154224 2.66457 1.64062 0.0344852 0.0293131 2024-04-22T08:30:58.946 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-04-22T08:30:58.232859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624664405,"num_objects":152517,"num_object_clones":0,"num_object_copies":305034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":152517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126119,"num_write_kb":504389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152644,"ondisk_log_size":152644,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2063324,"kb_used_data":1236740,"kb_used_omap":12,"kb_used_meta":826547,"kb_avail":281052196,"statfs":{"total":289910292480,"available":287797448704,"internally_reserved":0,"allocated":1266421760,"data_stored":1265724412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":846384238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,6,11,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17842176,"num_objects":4356,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-215,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":17424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":880640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011156"},"pg_stats":[{"pgid":"2.f","version":"37'9624","reported_seq":9686,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.896613+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:54.896613+0000","last_peered":"2024-04-22T08:30:54.896613+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:54.896613+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9624,"log_dups_size":0,"ondisk_log_size":9624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39419904,"num_objects":9624,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":38496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5830},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9627","reported_seq":9742,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.129898+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:55.129898+0000","last_peered":"2024-04-22T08:30:55.129898+0000","last_clean":"2024-04-22T08:30:55.129898+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:55.129898+0000","last_undegraded":"2024-04-22T08:30:55.129898+0000","last_fullsized":"2024-04-22T08:30:55.129898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39407616,"num_objects":9621,"num_object_clones":0,"num_object_copies":19242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9487","reported_seq":9610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.210182+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:55.210182+0000","last_peered":"2024-04-22T08:30:55.210182+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:55.210182+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":25256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9612","reported_seq":9642,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.136423+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:55.136423+0000","last_peered":"2024-04-22T08:30:55.136423+0000","last_clean":"2024-04-22T08:30:55.136423+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:55.136423+0000","last_undegraded":"2024-04-22T08:30:55.136423+0000","last_fullsized":"2024-04-22T08:30:55.136423+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":9612,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9512","reported_seq":9650,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:54.890971+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:54.890971+0000","last_peered":"2024-04-22T08:30:54.890971+0000","last_clean":"2024-04-22T08:30:54.890971+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:54.890971+0000","last_undegraded":"2024-04-22T08:30:54.890971+0000","last_fullsized":"2024-04-22T08:30:54.890971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932480,"num_objects":9505,"num_object_clones":0,"num_object_copies":19010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9558","reported_seq":9720,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.929974+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:54.929974+0000","last_peered":"2024-04-22T08:30:54.929974+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:54.929974+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129088,"num_objects":9553,"num_object_clones":0,"num_object_copies":19106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":25708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5674},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9453","reported_seq":17189,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:53.548573+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:53.548573+0000","last_peered":"2024-04-22T08:30:53.548573+0000","last_clean":"2024-04-22T08:30:53.548573+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:53.548573+0000","last_undegraded":"2024-04-22T08:30:53.548573+0000","last_fullsized":"2024-04-22T08:30:53.548573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38694912,"num_objects":9447,"num_object_clones":0,"num_object_copies":18894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":23284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9436","reported_seq":9580,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:54.885698+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:54.885698+0000","last_peered":"2024-04-22T08:30:54.885698+0000","last_clean":"2024-04-22T08:30:54.885698+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:54.885698+0000","last_undegraded":"2024-04-22T08:30:54.885698+0000","last_fullsized":"2024-04-22T08:30:54.885698+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":9434,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":24336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9689","reported_seq":9812,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.063197+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:55.063197+0000","last_peered":"2024-04-22T08:30:55.063197+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:55.063197+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39421138,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9599","reported_seq":9676,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.932065+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:54.932065+0000","last_peered":"2024-04-22T08:30:54.932065+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:54.932065+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9599,"log_dups_size":0,"ondisk_log_size":9599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39285970,"num_objects":9593,"num_object_clones":0,"num_object_copies":19186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":38368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9500","reported_seq":9645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.221786+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:55.221786+0000","last_peered":"2024-04-22T08:30:55.221786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:55.221786+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38891520,"num_objects":9495,"num_object_clones":0,"num_object_copies":18990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9495","reported_seq":11441,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:55.221726+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:55.221726+0000","last_peered":"2024-04-22T08:30:55.221726+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:55.221726+0000","last_undegraded":"2024-04-22T08:30:55.221726+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868178,"num_objects":9491,"num_object_clones":0,"num_object_copies":18982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":9491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":37959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":3870720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2858},{"shards":"2","objects":6633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9570","reported_seq":9649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.928350+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:54.928350+0000","last_peered":"2024-04-22T08:30:54.928350+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:54.928350+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191145,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9569,"num_write_kb":38273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5848},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9390","reported_seq":9565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.959654+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:54.959654+0000","last_peered":"2024-04-22T08:30:54.959654+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:54.959654+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38428672,"num_objects":9382,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":22484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5545},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9529","reported_seq":9604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.830845+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:54.830845+0000","last_peered":"2024-04-22T08:30:54.830845+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:54.830845+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015634,"num_objects":9527,"num_object_clones":0,"num_object_copies":19054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":38102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9563","reported_seq":9707,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.897965+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:54.897965+0000","last_peered":"2024-04-22T08:30:54.897965+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:54.897965+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39123364,"num_objects":9555,"num_object_clones":0,"num_object_copies":19110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9560,"num_write_kb":38211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5674},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":624664405,"num_objects":152517,"num_object_clones":0,"num_object_copies":305034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":152517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126119,"num_write_kb":504389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267163136,"data_stored":1267083119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152644,"ondisk_log_size":152644,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117043,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688492,"kb_used_data":426340,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93683348,"statfs":{"total":96636764160,"available":95931748352,"internally_reserved":0,"allocated":436572160,"data_stored":436360750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.45200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920805,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540744,"kb_used_data":292544,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93831096,"statfs":{"total":96636764160,"available":96083042304,"internally_reserved":0,"allocated":299565056,"data_stored":299284066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":254144204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.1079999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":15.989000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":20.149000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313271,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":834088,"kb_used_data":517856,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93537752,"statfs":{"total":96636764160,"available":95782658048,"internally_reserved":0,"allocated":530284544,"data_stored":530079596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":323808979},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531042304,"data_stored":531038825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299257856,"data_stored":299184797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":436862976,"data_stored":436859497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:58.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:30:59.783 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:30:59.786 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:30:59.829 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-04-22T08:30:58.232859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624664405,"num_objects":152517,"num_object_clones":0,"num_object_copies":305034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":152517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126119,"num_write_kb":504389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152644,"ondisk_log_size":152644,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2063324,"kb_used_data":1236740,"kb_used_omap":12,"kb_used_meta":826547,"kb_avail":281052196,"statfs":{"total":289910292480,"available":287797448704,"internally_reserved":0,"allocated":1266421760,"data_stored":1265724412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":846384238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,6,11,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17842176,"num_objects":4356,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-215,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":17424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":880640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011156"},"pg_stats":[{"pgid":"2.f","version":"37'9624","reported_seq":9686,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.896613+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:30:54.896613+0000","last_peered":"2024-04-22T08:30:54.896613+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:30:54.896613+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9624,"log_dups_size":0,"ondisk_log_size":9624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39419904,"num_objects":9624,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":38496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5830},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9627","reported_seq":9742,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.129898+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:55.129898+0000","last_peered":"2024-04-22T08:30:55.129898+0000","last_clean":"2024-04-22T08:30:55.129898+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:55.129898+0000","last_undegraded":"2024-04-22T08:30:55.129898+0000","last_fullsized":"2024-04-22T08:30:55.129898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39407616,"num_objects":9621,"num_object_clones":0,"num_object_copies":19242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9487","reported_seq":9610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.210182+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:55.210182+0000","last_peered":"2024-04-22T08:30:55.210182+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:55.210182+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":25256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9612","reported_seq":9642,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.136423+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:55.136423+0000","last_peered":"2024-04-22T08:30:55.136423+0000","last_clean":"2024-04-22T08:30:55.136423+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:55.136423+0000","last_undegraded":"2024-04-22T08:30:55.136423+0000","last_fullsized":"2024-04-22T08:30:55.136423+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":9612,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9512","reported_seq":9650,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:54.890971+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:54.890971+0000","last_peered":"2024-04-22T08:30:54.890971+0000","last_clean":"2024-04-22T08:30:54.890971+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:54.890971+0000","last_undegraded":"2024-04-22T08:30:54.890971+0000","last_fullsized":"2024-04-22T08:30:54.890971+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932480,"num_objects":9505,"num_object_clones":0,"num_object_copies":19010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9558","reported_seq":9720,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.929974+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:54.929974+0000","last_peered":"2024-04-22T08:30:54.929974+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:54.929974+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129088,"num_objects":9553,"num_object_clones":0,"num_object_copies":19106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":25708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5674},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9453","reported_seq":17189,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:53.548573+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:53.548573+0000","last_peered":"2024-04-22T08:30:53.548573+0000","last_clean":"2024-04-22T08:30:53.548573+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:53.548573+0000","last_undegraded":"2024-04-22T08:30:53.548573+0000","last_fullsized":"2024-04-22T08:30:53.548573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38694912,"num_objects":9447,"num_object_clones":0,"num_object_copies":18894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":23284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9436","reported_seq":9580,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:54.885698+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:54.885698+0000","last_peered":"2024-04-22T08:30:54.885698+0000","last_clean":"2024-04-22T08:30:54.885698+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:54.885698+0000","last_undegraded":"2024-04-22T08:30:54.885698+0000","last_fullsized":"2024-04-22T08:30:54.885698+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":9434,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":24336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9689","reported_seq":9812,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.063197+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:55.063197+0000","last_peered":"2024-04-22T08:30:55.063197+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:55.063197+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39421138,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9599","reported_seq":9676,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.932065+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:54.932065+0000","last_peered":"2024-04-22T08:30:54.932065+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:54.932065+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9599,"log_dups_size":0,"ondisk_log_size":9599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39285970,"num_objects":9593,"num_object_clones":0,"num_object_copies":19186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":38368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":67,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.144390+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:09.144390+0000","last_peered":"2024-04-22T08:30:09.144390+0000","last_clean":"2024-04-22T08:30:09.144390+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:09.144390+0000","last_undegraded":"2024-04-22T08:30:09.144390+0000","last_fullsized":"2024-04-22T08:30:09.144390+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9500","reported_seq":9645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.221786+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:55.221786+0000","last_peered":"2024-04-22T08:30:55.221786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:55.221786+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38891520,"num_objects":9495,"num_object_clones":0,"num_object_copies":18990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9495","reported_seq":11441,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:55.221726+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:55.221726+0000","last_peered":"2024-04-22T08:30:55.221726+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:55.221726+0000","last_undegraded":"2024-04-22T08:30:55.221726+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868178,"num_objects":9491,"num_object_clones":0,"num_object_copies":18982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":9491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":37959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":3870720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2858},{"shards":"2","objects":6633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9570","reported_seq":9649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.928350+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:54.928350+0000","last_peered":"2024-04-22T08:30:54.928350+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:54.928350+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191145,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9569,"num_write_kb":38273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5848},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9390","reported_seq":9565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.959654+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:30:54.959654+0000","last_peered":"2024-04-22T08:30:54.959654+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:30:54.959654+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38428672,"num_objects":9382,"num_object_clones":0,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":22484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5545},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9529","reported_seq":9604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.830845+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:54.830845+0000","last_peered":"2024-04-22T08:30:54.830845+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:54.830845+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39015634,"num_objects":9527,"num_object_clones":0,"num_object_copies":19054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":38102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9563","reported_seq":9707,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:54.897965+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:54.897965+0000","last_peered":"2024-04-22T08:30:54.897965+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:54.897965+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39123364,"num_objects":9555,"num_object_clones":0,"num_object_copies":19110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9560,"num_write_kb":38211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5674},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":624664405,"num_objects":152517,"num_object_clones":0,"num_object_copies":305034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":152517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126119,"num_write_kb":504389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267163136,"data_stored":1267083119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152644,"ondisk_log_size":152644,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117043,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688492,"kb_used_data":426340,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93683348,"statfs":{"total":96636764160,"available":95931748352,"internally_reserved":0,"allocated":436572160,"data_stored":436360750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.45200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920805,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540744,"kb_used_data":292544,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93831096,"statfs":{"total":96636764160,"available":96083042304,"internally_reserved":0,"allocated":299565056,"data_stored":299284066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":254144204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.313},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.1079999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":15.989000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":20.149000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313271,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":834088,"kb_used_data":517856,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93537752,"statfs":{"total":96636764160,"available":95782658048,"internally_reserved":0,"allocated":530284544,"data_stored":530079596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":323808979},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.52200000000000002,"15min":0.38500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.877,"15min":0.877},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.665,"5min":5.665,"15min":5.665},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531042304,"data_stored":531038825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299257856,"data_stored":299184797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":436862976,"data_stored":436859497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:30:59.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:30:59.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'9624", 'reported_seq': 9686, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.896613+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:30:54.896613+0000', 'last_peered': '2024-04-22T08:30:54.896613+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:30:54.896613+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9624, 'log_dups_size': 0, 'ondisk_log_size': 9624, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39419904, 'num_objects': 9624, 'num_object_clones': 0, 'num_object_copies': 19248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9624, 'num_write_kb': 38496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5830}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:30:59.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9487", 'reported_seq': 9610, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:55.210182+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:30:55.210182+0000', 'last_peered': '2024-04-22T08:30:55.210182+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:30:55.210182+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9487, 'log_dups_size': 0, 'ondisk_log_size': 9487, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38842368, 'num_objects': 9483, 'num_object_clones': 0, 'num_object_copies': 18966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6314, 'num_write_kb': 25256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:59.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9558", 'reported_seq': 9720, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.929974+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:54.929974+0000', 'last_peered': '2024-04-22T08:30:54.929974+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:54.929974+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9558, 'log_dups_size': 0, 'ondisk_log_size': 9558, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39129088, 'num_objects': 9553, 'num_object_clones': 0, 'num_object_copies': 19106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6427, 'num_write_kb': 25708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5674}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'9689", 'reported_seq': 9812, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:55.063197+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:30:55.063197+0000', 'last_peered': '2024-04-22T08:30:55.063197+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:30:55.063197+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 9689, 'log_dups_size': 0, 'ondisk_log_size': 9689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39421138, 'num_objects': 9626, 'num_object_clones': 0, 'num_object_copies': 19252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9629, 'num_write_kb': 38500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'9599", 'reported_seq': 9676, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.932065+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:54.932065+0000', 'last_peered': '2024-04-22T08:30:54.932065+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:54.932065+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9599, 'log_dups_size': 0, 'ondisk_log_size': 9599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39285970, 'num_objects': 9593, 'num_object_clones': 0, 'num_object_copies': 19186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9597, 'num_write_kb': 38368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5822}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'9500", 'reported_seq': 9645, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:55.221786+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:30:55.221786+0000', 'last_peered': '2024-04-22T08:30:55.221786+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:30:55.221786+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 9500, 'log_dups_size': 0, 'ondisk_log_size': 9500, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38891520, 'num_objects': 9495, 'num_object_clones': 0, 'num_object_copies': 18990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5954, 'num_write_kb': 23816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'9495", 'reported_seq': 11441, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:30:55.221726+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:30:55.221726+0000', 'last_peered': '2024-04-22T08:30:55.221726+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:30:55.221726+0000', 'last_undegraded': '2024-04-22T08:30:55.221726+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9495, 'log_dups_size': 0, 'ondisk_log_size': 9495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38868178, 'num_objects': 9491, 'num_object_clones': 0, 'num_object_copies': 18982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2858, 'num_objects_unfound': 0, 'num_objects_dirty': 9491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9493, 'num_write_kb': 37959, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 947, 'num_bytes_recovered': 3870720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2858}, {'shards': '2', 'objects': 6633}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:30:59.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'9570", 'reported_seq': 9649, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.928350+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:54.928350+0000', 'last_peered': '2024-04-22T08:30:54.928350+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:54.928350+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9570, 'log_dups_size': 0, 'ondisk_log_size': 9570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39191145, 'num_objects': 9569, 'num_object_clones': 0, 'num_object_copies': 19138, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9569, 'num_write_kb': 38273, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5848}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'9390", 'reported_seq': 9565, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.959654+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:30:54.959654+0000', 'last_peered': '2024-04-22T08:30:54.959654+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:30:54.959654+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 9390, 'log_dups_size': 0, 'ondisk_log_size': 9390, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38428672, 'num_objects': 9382, 'num_object_clones': 0, 'num_object_copies': 18764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5621, 'num_write_kb': 22484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5545}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9529", 'reported_seq': 9604, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.830845+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:54.830845+0000', 'last_peered': '2024-04-22T08:30:54.830845+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:54.830845+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9529, 'log_dups_size': 0, 'ondisk_log_size': 9529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39015634, 'num_objects': 9527, 'num_object_clones': 0, 'num_object_copies': 19054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9527, 'num_write_kb': 38102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5822}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:30:59.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9563", 'reported_seq': 9707, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:54.897965+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:54.897965+0000', 'last_peered': '2024-04-22T08:30:54.897965+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:54.897965+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9563, 'log_dups_size': 0, 'ondisk_log_size': 9563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39123364, 'num_objects': 9555, 'num_object_clones': 0, 'num_object_copies': 19110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9560, 'num_write_kb': 38211, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5674}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:30:59.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:30:59.904 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 227 20 154644 154624 2.65972 1.5625 0.0868157 0.0293649 2024-04-22T08:31:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:00.867 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/307762 objects degraded (12.347%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":295,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":153881,"data_bytes":630251349,"bytes_used":2143141888,"bytes_avail":287767150592,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":307762,"degraded_ratio":0.12346878432035144,"misplaced_objects":2858,"misplaced_total":307762,"misplaced_ratio":0.0092863966311630408,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":73318,"recovering_keys_per_sec":0,"num_objects_recovered":215,"num_bytes_recovered":880640,"num_keys_recovered":0,"write_bytes_sec":1950625,"read_op_per_sec":0,"write_op_per_sec":476},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:00.904 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 228 20 155063 155043 2.65523 1.63672 0.0462151 0.0294153 2024-04-22T08:31:00.962 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:00.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:01.826 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:01.827 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:01.911 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 229 20 155500 155480 2.651 1.70703 0.0556447 0.0294631 2024-04-22T08:31:01.920 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-22T08:31:00.233320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630251349,"num_objects":153881,"num_object_clones":0,"num_object_copies":307762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":153881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127483,"num_write_kb":509845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154008,"ondisk_log_size":154008,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092912,"kb_used_data":1249944,"kb_used_omap":12,"kb_used_meta":842931,"kb_avail":281022608,"statfs":{"total":289910292480,"available":287767150592,"internally_reserved":0,"allocated":1279942656,"data_stored":1279245308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":863161454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,17,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23429120,"num_objects":5720,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-215,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":22880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":880640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011082"},"pg_stats":[{"pgid":"2.f","version":"37'9756","reported_seq":9818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.010716+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:00.010716+0000","last_peered":"2024-04-22T08:31:00.010716+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:00.010716+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":39024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9627","reported_seq":9742,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.129898+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:30:55.129898+0000","last_peered":"2024-04-22T08:30:55.129898+0000","last_clean":"2024-04-22T08:30:55.129898+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:30:55.129898+0000","last_undegraded":"2024-04-22T08:30:55.129898+0000","last_fullsized":"2024-04-22T08:30:55.129898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39407616,"num_objects":9621,"num_object_clones":0,"num_object_copies":19242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9487","reported_seq":9610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.210182+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:30:55.210182+0000","last_peered":"2024-04-22T08:30:55.210182+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:30:55.210182+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":25256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9612","reported_seq":9642,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:55.136423+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:30:55.136423+0000","last_peered":"2024-04-22T08:30:55.136423+0000","last_clean":"2024-04-22T08:30:55.136423+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:30:55.136423+0000","last_undegraded":"2024-04-22T08:30:55.136423+0000","last_fullsized":"2024-04-22T08:30:55.136423+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":9612,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9651","reported_seq":9789,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.979523+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:59.979523+0000","last_peered":"2024-04-22T08:30:59.979523+0000","last_clean":"2024-04-22T08:30:59.979523+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:59.979523+0000","last_undegraded":"2024-04-22T08:30:59.979523+0000","last_fullsized":"2024-04-22T08:30:59.979523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":9644,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":26972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9697","reported_seq":9859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.966501+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:59.966501+0000","last_peered":"2024-04-22T08:30:59.966501+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:59.966501+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39698432,"num_objects":9692,"num_object_clones":0,"num_object_copies":19384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":26264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5813},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9584","reported_seq":17320,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:58.440696+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:30:58.440696+0000","last_peered":"2024-04-22T08:30:58.440696+0000","last_clean":"2024-04-22T08:30:58.440696+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:30:58.440696+0000","last_undegraded":"2024-04-22T08:30:58.440696+0000","last_fullsized":"2024-04-22T08:30:58.440696+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39231488,"num_objects":9578,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5952,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9566","reported_seq":9710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.949072+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:59.949072+0000","last_peered":"2024-04-22T08:30:59.949072+0000","last_clean":"2024-04-22T08:30:59.949072+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:59.949072+0000","last_undegraded":"2024-04-22T08:30:59.949072+0000","last_fullsized":"2024-04-22T08:30:59.949072+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39174144,"num_objects":9564,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6214,"num_write_kb":24856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9689","reported_seq":9812,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.063197+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:30:55.063197+0000","last_peered":"2024-04-22T08:30:55.063197+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:30:55.063197+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39421138,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9746","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.996077+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:59.996077+0000","last_peered":"2024-04-22T08:30:59.996077+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:59.996077+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39888082,"num_objects":9740,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9500","reported_seq":9645,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:55.221786+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:30:55.221786+0000","last_peered":"2024-04-22T08:30:55.221786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:30:55.221786+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38891520,"num_objects":9495,"num_object_clones":0,"num_object_copies":18990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9495","reported_seq":11441,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:30:55.221726+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:30:55.221726+0000","last_peered":"2024-04-22T08:30:55.221726+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:30:55.221726+0000","last_undegraded":"2024-04-22T08:30:55.221726+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868178,"num_objects":9491,"num_object_clones":0,"num_object_copies":18982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":9491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":37959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":3870720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2858},{"shards":"2","objects":6633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9707","reported_seq":9786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.938981+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:59.938981+0000","last_peered":"2024-04-22T08:30:59.938981+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:59.938981+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39752297,"num_objects":9706,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":38821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9534","reported_seq":9709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.007070+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:00.007070+0000","last_peered":"2024-04-22T08:31:00.007070+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:00.007070+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5765,"num_write_kb":23060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5689},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9658","reported_seq":9734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.957765+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:59.957765+0000","last_peered":"2024-04-22T08:30:59.957765+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:59.957765+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9658,"log_dups_size":0,"ondisk_log_size":9658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39544018,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":38618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5951},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9699","reported_seq":9843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.968073+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:59.968073+0000","last_peered":"2024-04-22T08:30:59.968073+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:59.968073+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39680420,"num_objects":9691,"num_object_clones":0,"num_object_copies":19382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9696,"num_write_kb":38755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5810},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":630251349,"num_objects":153881,"num_object_clones":0,"num_object_copies":307762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":153881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127483,"num_write_kb":509845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4748,"num_bytes_recovered":19374697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280872448,"data_stored":1280792431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154008,"ondisk_log_size":154008,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117043,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688492,"kb_used_data":426340,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93683348,"statfs":{"total":96636764160,"available":95931748352,"internally_reserved":0,"allocated":436572160,"data_stored":436360750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.45200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920806,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555832,"kb_used_data":299440,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93816008,"statfs":{"total":96636764160,"available":96067592192,"internally_reserved":0,"allocated":306626560,"data_stored":306345570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":262532812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.58899999999999997,"15min":0.52500000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1349999999999998,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:29:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.3160000000000001,"15min":1.3540000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.1210000000000004,"5min":8.6379999999999999,"15min":8.6379999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.1879999999999999,"15min":1.163},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.7300000000000004,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.66000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313272,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":848588,"kb_used_data":524164,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93523252,"statfs":{"total":96636764160,"available":95767810048,"internally_reserved":0,"allocated":536743936,"data_stored":536538988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":332197587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537612288,"data_stored":537608809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306397184,"data_stored":306324125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":436862976,"data_stored":436859497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:02.911 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 230 20 155934 155914 2.64685 1.69531 0.00664677 0.0295062 2024-04-22T08:31:03.911 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 231 20 156381 156361 2.64295 1.74609 0.0690395 0.0295514 2024-04-22T08:31:04.916 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 232 20 156823 156803 2.63895 1.72656 0.0616471 0.0295967 2024-04-22T08:31:04.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:05.467 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:05.483 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-04-22T08:31:04.234346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634203989,"num_objects":154846,"num_object_clones":0,"num_object_copies":309692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":154846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128448,"num_write_kb":513705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4847,"num_bytes_recovered":19780201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154973,"ondisk_log_size":154973,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2095608,"kb_used_data":1260828,"kb_used_omap":12,"kb_used_meta":834739,"kb_avail":281019912,"statfs":{"total":289910292480,"available":287764389888,"internally_reserved":0,"allocated":1291087872,"data_stored":1290390524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":854772846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,14,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18452480,"num_objects":4505,"num_object_clones":0,"num_object_copies":9010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-202,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":18020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005717"},"pg_stats":[{"pgid":"2.f","version":"37'9756","reported_seq":9818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.010716+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:00.010716+0000","last_peered":"2024-04-22T08:31:00.010716+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:00.010716+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":39024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9769","reported_seq":9884,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:00.261465+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:00.261465+0000","last_peered":"2024-04-22T08:31:00.261465+0000","last_clean":"2024-04-22T08:31:00.261465+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:00.261465+0000","last_undegraded":"2024-04-22T08:31:00.261465+0000","last_fullsized":"2024-04-22T08:31:00.261465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39989248,"num_objects":9763,"num_object_clones":0,"num_object_copies":19526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":27300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9620","reported_seq":9743,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.225510+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:00.225510+0000","last_peered":"2024-04-22T08:31:00.225510+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:00.225510+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9620,"log_dups_size":0,"ondisk_log_size":9620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":9616,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9731","reported_seq":9761,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:00.225485+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:00.225485+0000","last_peered":"2024-04-22T08:31:00.225485+0000","last_clean":"2024-04-22T08:31:00.225485+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:00.225485+0000","last_undegraded":"2024-04-22T08:31:00.225485+0000","last_fullsized":"2024-04-22T08:31:00.225485+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39858176,"num_objects":9731,"num_object_clones":0,"num_object_copies":19462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":38924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9651","reported_seq":9789,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.979523+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:59.979523+0000","last_peered":"2024-04-22T08:30:59.979523+0000","last_clean":"2024-04-22T08:30:59.979523+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:59.979523+0000","last_undegraded":"2024-04-22T08:30:59.979523+0000","last_fullsized":"2024-04-22T08:30:59.979523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":9644,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":26972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9697","reported_seq":9859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.966501+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:59.966501+0000","last_peered":"2024-04-22T08:30:59.966501+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:59.966501+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39698432,"num_objects":9692,"num_object_clones":0,"num_object_copies":19384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":26264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5813},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9720","reported_seq":17456,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:03.601520+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:03.601520+0000","last_peered":"2024-04-22T08:31:03.601520+0000","last_clean":"2024-04-22T08:31:03.601520+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:03.601520+0000","last_undegraded":"2024-04-22T08:31:03.601520+0000","last_fullsized":"2024-04-22T08:31:03.601520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9720,"log_dups_size":0,"ondisk_log_size":9720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39788544,"num_objects":9714,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":24352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9566","reported_seq":9710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.949072+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:59.949072+0000","last_peered":"2024-04-22T08:30:59.949072+0000","last_clean":"2024-04-22T08:30:59.949072+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:59.949072+0000","last_undegraded":"2024-04-22T08:30:59.949072+0000","last_fullsized":"2024-04-22T08:30:59.949072+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39174144,"num_objects":9564,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6214,"num_write_kb":24856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9836","reported_seq":9958,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.260712+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:00.260712+0000","last_peered":"2024-04-22T08:31:00.260712+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:00.260712+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023250,"num_objects":9773,"num_object_clones":0,"num_object_copies":19546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9776,"num_write_kb":39088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9746","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.996077+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:59.996077+0000","last_peered":"2024-04-22T08:30:59.996077+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:59.996077+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39888082,"num_objects":9740,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9642","reported_seq":9788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.163689+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:00.163689+0000","last_peered":"2024-04-22T08:31:00.163689+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:00.163689+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39473152,"num_objects":9637,"num_object_clones":0,"num_object_copies":19274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5887}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9641","reported_seq":11784,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:00.255655+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:00.255655+0000","last_peered":"2024-04-22T08:31:00.255655+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:00.255655+0000","last_undegraded":"2024-04-22T08:31:00.255655+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466194,"num_objects":9637,"num_object_clones":0,"num_object_copies":19274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":9637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":38543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2759},{"shards":"2","objects":6878}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9707","reported_seq":9786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.938981+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:59.938981+0000","last_peered":"2024-04-22T08:30:59.938981+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:59.938981+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39752297,"num_objects":9706,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":38821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9534","reported_seq":9709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.007070+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:00.007070+0000","last_peered":"2024-04-22T08:31:00.007070+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:00.007070+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5765,"num_write_kb":23060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5689},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9658","reported_seq":9734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.957765+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:59.957765+0000","last_peered":"2024-04-22T08:30:59.957765+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:59.957765+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9658,"log_dups_size":0,"ondisk_log_size":9658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39544018,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":38618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5951},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9699","reported_seq":9843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.968073+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:59.968073+0000","last_peered":"2024-04-22T08:30:59.968073+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:59.968073+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39680420,"num_objects":9691,"num_object_clones":0,"num_object_copies":19382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9696,"num_write_kb":38755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5810},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634203989,"num_objects":154846,"num_object_clones":0,"num_object_copies":309692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":154846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128448,"num_write_kb":513705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4847,"num_bytes_recovered":19780201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292361728,"data_stored":1292281711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154973,"ondisk_log_size":154973,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117044,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696832,"kb_used_data":430580,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93675008,"statfs":{"total":96636764160,"available":95923208192,"internally_reserved":0,"allocated":440913920,"data_stored":440702510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.45200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920807,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550188,"kb_used_data":306084,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93821652,"statfs":{"total":96636764160,"available":96073371648,"internally_reserved":0,"allocated":313430016,"data_stored":313149026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":249949900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.57399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313272,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":848588,"kb_used_data":524164,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93523252,"statfs":{"total":96636764160,"available":95767810048,"internally_reserved":0,"allocated":536743936,"data_stored":536538988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":332197587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537612288,"data_stored":537608809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313430016,"data_stored":313356957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":441319424,"data_stored":441315945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:05.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:05.916 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 233 20 157157 157137 2.63322 1.30469 0.102965 0.0296597 2024-04-22T08:31:06.211 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:06.211 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:06.249 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-04-22T08:31:04.234346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634203989,"num_objects":154846,"num_object_clones":0,"num_object_copies":309692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":154846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128448,"num_write_kb":513705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4847,"num_bytes_recovered":19780201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154973,"ondisk_log_size":154973,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2095608,"kb_used_data":1260828,"kb_used_omap":12,"kb_used_meta":834739,"kb_avail":281019912,"statfs":{"total":289910292480,"available":287764389888,"internally_reserved":0,"allocated":1291087872,"data_stored":1290390524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":854772846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,14,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18452480,"num_objects":4505,"num_object_clones":0,"num_object_copies":9010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-202,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":18020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005717"},"pg_stats":[{"pgid":"2.f","version":"37'9756","reported_seq":9818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.010716+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:00.010716+0000","last_peered":"2024-04-22T08:31:00.010716+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:00.010716+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":39024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9769","reported_seq":9884,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:00.261465+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:00.261465+0000","last_peered":"2024-04-22T08:31:00.261465+0000","last_clean":"2024-04-22T08:31:00.261465+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:00.261465+0000","last_undegraded":"2024-04-22T08:31:00.261465+0000","last_fullsized":"2024-04-22T08:31:00.261465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39989248,"num_objects":9763,"num_object_clones":0,"num_object_copies":19526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":27300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9620","reported_seq":9743,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.225510+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:00.225510+0000","last_peered":"2024-04-22T08:31:00.225510+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:00.225510+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9620,"log_dups_size":0,"ondisk_log_size":9620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":9616,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9731","reported_seq":9761,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:00.225485+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:00.225485+0000","last_peered":"2024-04-22T08:31:00.225485+0000","last_clean":"2024-04-22T08:31:00.225485+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:00.225485+0000","last_undegraded":"2024-04-22T08:31:00.225485+0000","last_fullsized":"2024-04-22T08:31:00.225485+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39858176,"num_objects":9731,"num_object_clones":0,"num_object_copies":19462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":38924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9651","reported_seq":9789,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.979523+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:30:59.979523+0000","last_peered":"2024-04-22T08:30:59.979523+0000","last_clean":"2024-04-22T08:30:59.979523+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:30:59.979523+0000","last_undegraded":"2024-04-22T08:30:59.979523+0000","last_fullsized":"2024-04-22T08:30:59.979523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39501824,"num_objects":9644,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":26972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9697","reported_seq":9859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.966501+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:30:59.966501+0000","last_peered":"2024-04-22T08:30:59.966501+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:30:59.966501+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39698432,"num_objects":9692,"num_object_clones":0,"num_object_copies":19384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":26264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5813},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9720","reported_seq":17456,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:03.601520+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:03.601520+0000","last_peered":"2024-04-22T08:31:03.601520+0000","last_clean":"2024-04-22T08:31:03.601520+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:03.601520+0000","last_undegraded":"2024-04-22T08:31:03.601520+0000","last_fullsized":"2024-04-22T08:31:03.601520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9720,"log_dups_size":0,"ondisk_log_size":9720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39788544,"num_objects":9714,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":24352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9566","reported_seq":9710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:59.949072+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:30:59.949072+0000","last_peered":"2024-04-22T08:30:59.949072+0000","last_clean":"2024-04-22T08:30:59.949072+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:30:59.949072+0000","last_undegraded":"2024-04-22T08:30:59.949072+0000","last_fullsized":"2024-04-22T08:30:59.949072+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39174144,"num_objects":9564,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6214,"num_write_kb":24856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9836","reported_seq":9958,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.260712+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:00.260712+0000","last_peered":"2024-04-22T08:31:00.260712+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:00.260712+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023250,"num_objects":9773,"num_object_clones":0,"num_object_copies":19546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9776,"num_write_kb":39088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":5885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9746","reported_seq":9823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.996077+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:30:59.996077+0000","last_peered":"2024-04-22T08:30:59.996077+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:30:59.996077+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39888082,"num_objects":9740,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9642","reported_seq":9788,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.163689+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:00.163689+0000","last_peered":"2024-04-22T08:31:00.163689+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:00.163689+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39473152,"num_objects":9637,"num_object_clones":0,"num_object_copies":19274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":5887}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9641","reported_seq":11784,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:00.255655+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:00.255655+0000","last_peered":"2024-04-22T08:31:00.255655+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:00.255655+0000","last_undegraded":"2024-04-22T08:31:00.255655+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466194,"num_objects":9637,"num_object_clones":0,"num_object_copies":19274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":9637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":38543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2759},{"shards":"2","objects":6878}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9707","reported_seq":9786,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.938981+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:30:59.938981+0000","last_peered":"2024-04-22T08:30:59.938981+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:30:59.938981+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39752297,"num_objects":9706,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":38821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9534","reported_seq":9709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:00.007070+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:00.007070+0000","last_peered":"2024-04-22T08:31:00.007070+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:00.007070+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5765,"num_write_kb":23060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5689},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9658","reported_seq":9734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.957765+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:30:59.957765+0000","last_peered":"2024-04-22T08:30:59.957765+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:30:59.957765+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9658,"log_dups_size":0,"ondisk_log_size":9658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39544018,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":38618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5951},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9699","reported_seq":9843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:30:59.968073+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:30:59.968073+0000","last_peered":"2024-04-22T08:30:59.968073+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:30:59.968073+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39680420,"num_objects":9691,"num_object_clones":0,"num_object_copies":19382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9696,"num_write_kb":38755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5810},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634203989,"num_objects":154846,"num_object_clones":0,"num_object_copies":309692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2759,"num_objects_unfound":0,"num_objects_dirty":154846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128448,"num_write_kb":513705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4847,"num_bytes_recovered":19780201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292361728,"data_stored":1292281711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154973,"ondisk_log_size":154973,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117044,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696832,"kb_used_data":430580,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93675008,"statfs":{"total":96636764160,"available":95923208192,"internally_reserved":0,"allocated":440913920,"data_stored":440702510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":5.173,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.63300000000000001,"15min":0.59399999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.7309999999999999,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.45200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920807,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550188,"kb_used_data":306084,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93821652,"statfs":{"total":96636764160,"available":96073371648,"internally_reserved":0,"allocated":313430016,"data_stored":313149026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":249949900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.57399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313272,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":848588,"kb_used_data":524164,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93523252,"statfs":{"total":96636764160,"available":95767810048,"internally_reserved":0,"allocated":536743936,"data_stored":536538988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":332197587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537612288,"data_stored":537608809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313430016,"data_stored":313356957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":441319424,"data_stored":441315945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'9756", 'reported_seq': 9818, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:00.010716+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:00.010716+0000', 'last_peered': '2024-04-22T08:31:00.010716+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:00.010716+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9756, 'log_dups_size': 0, 'ondisk_log_size': 9756, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39960576, 'num_objects': 9756, 'num_object_clones': 0, 'num_object_copies': 19512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9756, 'num_write_kb': 39024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5962}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9620", 'reported_seq': 9743, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:00.225510+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:00.225510+0000', 'last_peered': '2024-04-22T08:31:00.225510+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:00.225510+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9620, 'log_dups_size': 0, 'ondisk_log_size': 9620, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39387136, 'num_objects': 9616, 'num_object_clones': 0, 'num_object_copies': 19232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 25788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 5861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9697", 'reported_seq': 9859, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:59.966501+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:30:59.966501+0000', 'last_peered': '2024-04-22T08:30:59.966501+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:30:59.966501+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9697, 'log_dups_size': 0, 'ondisk_log_size': 9697, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39698432, 'num_objects': 9692, 'num_object_clones': 0, 'num_object_copies': 19384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6566, 'num_write_kb': 26264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5813}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'9836", 'reported_seq': 9958, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:00.260712+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:00.260712+0000', 'last_peered': '2024-04-22T08:31:00.260712+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:00.260712+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 9836, 'log_dups_size': 0, 'ondisk_log_size': 9836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40023250, 'num_objects': 9773, 'num_object_clones': 0, 'num_object_copies': 19546, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9776, 'num_write_kb': 39088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 5885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'9746", 'reported_seq': 9823, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:59.996077+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:30:59.996077+0000', 'last_peered': '2024-04-22T08:30:59.996077+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:30:59.996077+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9746, 'log_dups_size': 0, 'ondisk_log_size': 9746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39888082, 'num_objects': 9740, 'num_object_clones': 0, 'num_object_copies': 19480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9744, 'num_write_kb': 38956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5969}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'9642", 'reported_seq': 9788, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:00.163689+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:00.163689+0000', 'last_peered': '2024-04-22T08:31:00.163689+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:00.163689+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 9642, 'log_dups_size': 0, 'ondisk_log_size': 9642, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39473152, 'num_objects': 9637, 'num_object_clones': 0, 'num_object_copies': 19274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6096, 'num_write_kb': 24384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 5887}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'9641", 'reported_seq': 11784, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:00.255655+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:00.255655+0000', 'last_peered': '2024-04-22T08:31:00.255655+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:00.255655+0000', 'last_undegraded': '2024-04-22T08:31:00.255655+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9641, 'log_dups_size': 0, 'ondisk_log_size': 9641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39466194, 'num_objects': 9637, 'num_object_clones': 0, 'num_object_copies': 19274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2759, 'num_objects_unfound': 0, 'num_objects_dirty': 9637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9639, 'num_write_kb': 38543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1046, 'num_bytes_recovered': 4276224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2759}, {'shards': '2', 'objects': 6878}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:06.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'9707", 'reported_seq': 9786, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:59.938981+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:30:59.938981+0000', 'last_peered': '2024-04-22T08:30:59.938981+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:30:59.938981+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9707, 'log_dups_size': 0, 'ondisk_log_size': 9707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39752297, 'num_objects': 9706, 'num_object_clones': 0, 'num_object_copies': 19412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9706, 'num_write_kb': 38821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5985}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'9534", 'reported_seq': 9709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:00.007070+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:00.007070+0000', 'last_peered': '2024-04-22T08:31:00.007070+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:00.007070+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 9534, 'log_dups_size': 0, 'ondisk_log_size': 9534, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39018496, 'num_objects': 9526, 'num_object_clones': 0, 'num_object_copies': 19052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5765, 'num_write_kb': 23060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5689}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9658", 'reported_seq': 9734, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:59.957765+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:30:59.957765+0000', 'last_peered': '2024-04-22T08:30:59.957765+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:30:59.957765+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9658, 'log_dups_size': 0, 'ondisk_log_size': 9658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39544018, 'num_objects': 9656, 'num_object_clones': 0, 'num_object_copies': 19312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9656, 'num_write_kb': 38618, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5951}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:06.253 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9699", 'reported_seq': 9843, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:30:59.968073+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:30:59.968073+0000', 'last_peered': '2024-04-22T08:30:59.968073+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:30:59.968073+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9699, 'log_dups_size': 0, 'ondisk_log_size': 9699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39680420, 'num_objects': 9691, 'num_object_clones': 0, 'num_object_copies': 19382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9696, 'num_write_kb': 38755, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5810}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:06.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:06.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 234 20 157547 157527 2.62848 1.52344 0.0358145 0.0297121 2024-04-22T08:31:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:07.126 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/313782 objects degraded (12.110%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":302,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":156891,"data_bytes":642580309,"bytes_used":2166677504,"bytes_avail":287743614976,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":313782,"degraded_ratio":0.12109999936261481,"misplaced_objects":2652,"misplaced_total":313782,"misplaced_ratio":0.0084517276325601857,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":105422,"recovering_keys_per_sec":0,"num_objects_recovered":309,"num_bytes_recovered":1265664,"num_keys_recovered":0,"write_bytes_sec":2179071,"read_op_per_sec":0,"write_op_per_sec":531},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:07.219 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:07.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:07.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 235 20 157948 157928 2.62396 1.56641 0.0889539 0.0297647 2024-04-22T08:31:07.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:07.943 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:07.992 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-04-22T08:31:06.234815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642580309,"num_objects":156891,"num_object_clones":0,"num_object_copies":313782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":156891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130493,"num_write_kb":521885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":20218473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157018,"ondisk_log_size":157018,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2115896,"kb_used_data":1272928,"kb_used_omap":12,"kb_used_meta":842931,"kb_avail":280999624,"statfs":{"total":289910292480,"available":287743614976,"internally_reserved":0,"allocated":1303478272,"data_stored":1302780924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":863161454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,6,9,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26161152,"num_objects":6387,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-309,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":25548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":1265664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005643"},"pg_stats":[{"pgid":"2.f","version":"37'9889","reported_seq":9952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.148947+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:05.148947+0000","last_peered":"2024-04-22T08:31:05.148947+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:05.148947+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40505344,"num_objects":9889,"num_object_clones":0,"num_object_copies":19778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":39556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6095},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9918","reported_seq":10033,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.259865+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:05.259865+0000","last_peered":"2024-04-22T08:31:05.259865+0000","last_clean":"2024-04-22T08:31:05.259865+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:05.259865+0000","last_undegraded":"2024-04-22T08:31:05.259865+0000","last_fullsized":"2024-04-22T08:31:05.259865+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40599552,"num_objects":9912,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9745","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.277120+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:05.277120+0000","last_peered":"2024-04-22T08:31:05.277120+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:05.277120+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39899136,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":26288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5986}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9865","reported_seq":9895,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.207547+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:05.207547+0000","last_peered":"2024-04-22T08:31:05.207547+0000","last_clean":"2024-04-22T08:31:05.207547+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:05.207547+0000","last_undegraded":"2024-04-22T08:31:05.207547+0000","last_fullsized":"2024-04-22T08:31:05.207547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40407040,"num_objects":9865,"num_object_clones":0,"num_object_copies":19730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":39460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9795","reported_seq":9933,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.048650+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:05.048650+0000","last_peered":"2024-04-22T08:31:05.048650+0000","last_clean":"2024-04-22T08:31:05.048650+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:05.048650+0000","last_undegraded":"2024-04-22T08:31:05.048650+0000","last_fullsized":"2024-04-22T08:31:05.048650+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9795,"log_dups_size":0,"ondisk_log_size":9795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40091648,"num_objects":9788,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9830","reported_seq":9992,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.014563+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:05.014563+0000","last_peered":"2024-04-22T08:31:05.014563+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:05.014563+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40243200,"num_objects":9825,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":26796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5946},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9720","reported_seq":17456,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:03.601520+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:03.601520+0000","last_peered":"2024-04-22T08:31:03.601520+0000","last_clean":"2024-04-22T08:31:03.601520+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:03.601520+0000","last_undegraded":"2024-04-22T08:31:03.601520+0000","last_fullsized":"2024-04-22T08:31:03.601520+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9720,"log_dups_size":0,"ondisk_log_size":9720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39788544,"num_objects":9714,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":24352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9706","reported_seq":9850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.031678+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:05.031678+0000","last_peered":"2024-04-22T08:31:05.031678+0000","last_clean":"2024-04-22T08:31:05.031678+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:05.031678+0000","last_undegraded":"2024-04-22T08:31:05.031678+0000","last_fullsized":"2024-04-22T08:31:05.031678+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747584,"num_objects":9704,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6354,"num_write_kb":25416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9965","reported_seq":10088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.263344+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:05.263344+0000","last_peered":"2024-04-22T08:31:05.263344+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:05.263344+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40551634,"num_objects":9902,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9905,"num_write_kb":39604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":6014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'9878","reported_seq":9956,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.063149+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:05.063149+0000","last_peered":"2024-04-22T08:31:05.063149+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:05.063149+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40428754,"num_objects":9872,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9876,"num_write_kb":39484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6101},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9784","reported_seq":9930,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.169348+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:05.169348+0000","last_peered":"2024-04-22T08:31:05.169348+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:05.169348+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9784,"log_dups_size":0,"ondisk_log_size":9784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40054784,"num_objects":9779,"num_object_clones":0,"num_object_copies":19558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6238,"num_write_kb":24952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":6029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9771","reported_seq":12130,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:05.305144+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:05.305144+0000","last_peered":"2024-04-22T08:31:05.305144+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:05.305144+0000","last_undegraded":"2024-04-22T08:31:05.305144+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39998674,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":39063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":4714496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2652},{"shards":"2","objects":7115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'9857","reported_seq":9936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.060553+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:05.060553+0000","last_peered":"2024-04-22T08:31:05.060553+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:05.060553+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9857,"log_dups_size":0,"ondisk_log_size":9857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40366697,"num_objects":9856,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9856,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6135},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9667","reported_seq":9842,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.071251+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:05.071251+0000","last_peered":"2024-04-22T08:31:05.071251+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:05.071251+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39563264,"num_objects":9659,"num_object_clones":0,"num_object_copies":19318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":23592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9814","reported_seq":9890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:04.948541+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:04.948541+0000","last_peered":"2024-04-22T08:31:04.948541+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:04.948541+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40182994,"num_objects":9812,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":39242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6107},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9814","reported_seq":9958,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.122335+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:05.122335+0000","last_peered":"2024-04-22T08:31:05.122335+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:05.122335+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40151460,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":39215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5925},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":642580309,"num_objects":156891,"num_object_clones":0,"num_object_copies":313782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":156891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130493,"num_write_kb":521885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":20218473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1303703552,"data_stored":1303623535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157018,"ondisk_log_size":157018,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117045,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705296,"kb_used_data":434952,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93666544,"statfs":{"total":96636764160,"available":95914541056,"internally_reserved":0,"allocated":445390848,"data_stored":445179438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":1.0389999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920807,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550188,"kb_used_data":306084,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93821652,"statfs":{"total":96636764160,"available":96073371648,"internally_reserved":0,"allocated":313430016,"data_stored":313149026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":249949900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.57399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313273,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":860412,"kb_used_data":531892,"kb_used_omap":4,"kb_used_meta":328507,"kb_avail":93511428,"statfs":{"total":96636764160,"available":95755702272,"internally_reserved":0,"allocated":544657408,"data_stored":544452460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":336391891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.437},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":544489472,"data_stored":544485993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313430016,"data_stored":313356957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445784064,"data_stored":445780585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:08.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 236 20 158386 158366 2.62009 1.71094 0.00641682 0.0298061 2024-04-22T08:31:09.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 237 20 158819 158799 2.61618 1.69141 0.0487229 0.0298543 2024-04-22T08:31:10.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 238 20 159258 159238 2.61239 1.71484 0.0374061 0.0298988 2024-04-22T08:31:10.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:11.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout: 239 20 159672 159652 2.60823 1.61719 0.0197224 0.0299443 2024-04-22T08:31:11.930 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:11.931 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-04-22T08:31:10.235774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647917397,"num_objects":158194,"num_object_clones":0,"num_object_copies":316388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":158194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131796,"num_write_kb":527097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":20218473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158321,"ondisk_log_size":158321,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2128548,"kb_used_data":1285580,"kb_used_omap":12,"kb_used_meta":842931,"kb_avail":280986972,"statfs":{"total":289910292480,"available":287730659328,"internally_reserved":0,"allocated":1316433920,"data_stored":1315736572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":863161454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,5,8,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23252992,"num_objects":5677,"num_object_clones":0,"num_object_copies":11354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5677,"num_write_kb":22708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002915"},"pg_stats":[{"pgid":"2.f","version":"37'10028","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.178304+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:10.178304+0000","last_peered":"2024-04-22T08:31:10.178304+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:10.178304+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41074688,"num_objects":10028,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":40112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6234},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'9918","reported_seq":10033,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.259865+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:05.259865+0000","last_peered":"2024-04-22T08:31:05.259865+0000","last_clean":"2024-04-22T08:31:05.259865+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:05.259865+0000","last_undegraded":"2024-04-22T08:31:05.259865+0000","last_fullsized":"2024-04-22T08:31:05.259865+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40599552,"num_objects":9912,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9745","reported_seq":9868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.277120+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:05.277120+0000","last_peered":"2024-04-22T08:31:05.277120+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:05.277120+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39899136,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":26288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":5986}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9865","reported_seq":9895,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:05.207547+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:05.207547+0000","last_peered":"2024-04-22T08:31:05.207547+0000","last_clean":"2024-04-22T08:31:05.207547+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:05.207547+0000","last_undegraded":"2024-04-22T08:31:05.207547+0000","last_fullsized":"2024-04-22T08:31:05.207547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40407040,"num_objects":9865,"num_object_clones":0,"num_object_copies":19730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":39460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9910","reported_seq":10048,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.149502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:10.149502+0000","last_peered":"2024-04-22T08:31:10.149502+0000","last_clean":"2024-04-22T08:31:10.149502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:10.149502+0000","last_undegraded":"2024-04-22T08:31:10.149502+0000","last_fullsized":"2024-04-22T08:31:10.149502+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7002,"num_write_kb":28008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9962","reported_seq":10124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.111784+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:10.111784+0000","last_peered":"2024-04-22T08:31:10.111784+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:10.111784+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40783872,"num_objects":9957,"num_object_clones":0,"num_object_copies":19914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":27324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6078},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9833","reported_seq":17569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:08.643506+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:08.643506+0000","last_peered":"2024-04-22T08:31:08.643506+0000","last_clean":"2024-04-22T08:31:08.643506+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:08.643506+0000","last_undegraded":"2024-04-22T08:31:08.643506+0000","last_fullsized":"2024-04-22T08:31:08.643506+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40251392,"num_objects":9827,"num_object_clones":0,"num_object_copies":19654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":24804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9823","reported_seq":9968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:09.989620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:09.989620+0000","last_peered":"2024-04-22T08:31:09.989620+0000","last_clean":"2024-04-22T08:31:09.989620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:09.989620+0000","last_undegraded":"2024-04-22T08:31:09.989620+0000","last_fullsized":"2024-04-22T08:31:09.989620+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40226816,"num_objects":9821,"num_object_clones":0,"num_object_copies":19642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":25884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'9965","reported_seq":10088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.263344+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:05.263344+0000","last_peered":"2024-04-22T08:31:05.263344+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:05.263344+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40551634,"num_objects":9902,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9905,"num_write_kb":39604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":6014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10015","reported_seq":10093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.102091+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:10.102091+0000","last_peered":"2024-04-22T08:31:10.102091+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:10.102091+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989906,"num_objects":10009,"num_object_clones":0,"num_object_copies":20018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":40032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9784","reported_seq":9930,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:05.169348+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:05.169348+0000","last_peered":"2024-04-22T08:31:05.169348+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:05.169348+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9784,"log_dups_size":0,"ondisk_log_size":9784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40054784,"num_objects":9779,"num_object_clones":0,"num_object_copies":19558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6238,"num_write_kb":24952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":6029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9771","reported_seq":12130,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:05.305144+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:05.305144+0000","last_peered":"2024-04-22T08:31:05.305144+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:05.305144+0000","last_undegraded":"2024-04-22T08:31:05.305144+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39998674,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":39063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":4714496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2652},{"shards":"2","objects":7115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10009","reported_seq":10088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.072111+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:10.072111+0000","last_peered":"2024-04-22T08:31:10.072111+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:10.072111+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989289,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9802","reported_seq":9977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.149548+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:10.149548+0000","last_peered":"2024-04-22T08:31:10.149548+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:10.149548+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40116224,"num_objects":9794,"num_object_clones":0,"num_object_copies":19588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":24132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5957},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9945","reported_seq":10019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.152886+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:10.152886+0000","last_peered":"2024-04-22T08:31:10.152886+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:10.152886+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719570,"num_objects":9943,"num_object_clones":0,"num_object_copies":19886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9946","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.095479+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:10.095479+0000","last_peered":"2024-04-22T08:31:10.095479+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:10.095479+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40692132,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6057},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":647917397,"num_objects":158194,"num_object_clones":0,"num_object_copies":316388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2652,"num_objects_unfound":0,"num_objects_dirty":158194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131796,"num_write_kb":527097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4954,"num_bytes_recovered":20218473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316900864,"data_stored":1316820847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158321,"ondisk_log_size":158321,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117045,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705296,"kb_used_data":434952,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93666544,"statfs":{"total":96636764160,"available":95914541056,"internally_reserved":0,"allocated":445390848,"data_stored":445179438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":1.0389999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920808,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564840,"kb_used_data":312544,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93807000,"statfs":{"total":96636764160,"available":96058368000,"internally_reserved":0,"allocated":320045056,"data_stored":319764066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":258338508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.3410000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.23100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.0760000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.2290000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313274,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":858412,"kb_used_data":538084,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93513428,"statfs":{"total":96636764160,"available":95757750272,"internally_reserved":0,"allocated":550998016,"data_stored":550793068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328003283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":550842368,"data_stored":550838889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320274432,"data_stored":320201373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445784064,"data_stored":445780585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:11.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:12.822 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:12.842 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-22T08:31:12.236277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650940245,"num_objects":158932,"num_object_clones":0,"num_object_copies":317864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":158932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132534,"num_write_kb":530049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":20742761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2136496,"kb_used_data":1289432,"kb_used_omap":12,"kb_used_meta":847027,"kb_avail":280979024,"statfs":{"total":289910292480,"available":287722520576,"internally_reserved":0,"allocated":1320378368,"data_stored":1319681020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":867355758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,6,7,9,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20688896,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-334,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":20204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002957"},"pg_stats":[{"pgid":"2.f","version":"37'10028","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.178304+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:10.178304+0000","last_peered":"2024-04-22T08:31:10.178304+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:10.178304+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41074688,"num_objects":10028,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":40112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6234},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10056","reported_seq":10171,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.317016+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:10.317016+0000","last_peered":"2024-04-22T08:31:10.317016+0000","last_clean":"2024-04-22T08:31:10.317016+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:10.317016+0000","last_undegraded":"2024-04-22T08:31:10.317016+0000","last_fullsized":"2024-04-22T08:31:10.317016+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41164800,"num_objects":10050,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":28448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9861","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.301599+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:10.301599+0000","last_peered":"2024-04-22T08:31:10.301599+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:10.301599+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9861,"log_dups_size":0,"ondisk_log_size":9861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40374272,"num_objects":9857,"num_object_clones":0,"num_object_copies":19714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":6102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9985","reported_seq":10015,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.352793+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:10.352793+0000","last_peered":"2024-04-22T08:31:10.352793+0000","last_clean":"2024-04-22T08:31:10.352793+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:10.352793+0000","last_undegraded":"2024-04-22T08:31:10.352793+0000","last_fullsized":"2024-04-22T08:31:10.352793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40898560,"num_objects":9985,"num_object_clones":0,"num_object_copies":19970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":39940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9910","reported_seq":10048,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.149502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:10.149502+0000","last_peered":"2024-04-22T08:31:10.149502+0000","last_clean":"2024-04-22T08:31:10.149502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:10.149502+0000","last_undegraded":"2024-04-22T08:31:10.149502+0000","last_fullsized":"2024-04-22T08:31:10.149502+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7002,"num_write_kb":28008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9962","reported_seq":10124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.111784+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:10.111784+0000","last_peered":"2024-04-22T08:31:10.111784+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:10.111784+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40783872,"num_objects":9957,"num_object_clones":0,"num_object_copies":19914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":27324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6078},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9833","reported_seq":17569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:08.643506+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:08.643506+0000","last_peered":"2024-04-22T08:31:08.643506+0000","last_clean":"2024-04-22T08:31:08.643506+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:08.643506+0000","last_undegraded":"2024-04-22T08:31:08.643506+0000","last_fullsized":"2024-04-22T08:31:08.643506+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40251392,"num_objects":9827,"num_object_clones":0,"num_object_copies":19654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":24804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9823","reported_seq":9968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:09.989620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:09.989620+0000","last_peered":"2024-04-22T08:31:09.989620+0000","last_clean":"2024-04-22T08:31:09.989620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:09.989620+0000","last_undegraded":"2024-04-22T08:31:09.989620+0000","last_fullsized":"2024-04-22T08:31:09.989620+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40226816,"num_objects":9821,"num_object_clones":0,"num_object_copies":19642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":25884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10082","reported_seq":10205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.315786+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:10.315786+0000","last_peered":"2024-04-22T08:31:10.315786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:10.315786+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41030866,"num_objects":10019,"num_object_clones":0,"num_object_copies":20038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10022,"num_write_kb":40072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":6131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10015","reported_seq":10093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.102091+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:10.102091+0000","last_peered":"2024-04-22T08:31:10.102091+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:10.102091+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989906,"num_objects":10009,"num_object_clones":0,"num_object_copies":20018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":40032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9905","reported_seq":10050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.356342+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:10.356342+0000","last_peered":"2024-04-22T08:31:10.356342+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:10.356342+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40550400,"num_objects":9900,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":25436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":6150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9897","reported_seq":12512,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:10.303582+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:10.303582+0000","last_peered":"2024-04-22T08:31:10.303582+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:10.303582+0000","last_undegraded":"2024-04-22T08:31:10.303582+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40514770,"num_objects":9893,"num_object_clones":0,"num_object_copies":19786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":9893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":39567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1281,"num_bytes_recovered":5238784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2524},{"shards":"2","objects":7369}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10009","reported_seq":10088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.072111+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:10.072111+0000","last_peered":"2024-04-22T08:31:10.072111+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:10.072111+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989289,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9802","reported_seq":9977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.149548+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:10.149548+0000","last_peered":"2024-04-22T08:31:10.149548+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:10.149548+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40116224,"num_objects":9794,"num_object_clones":0,"num_object_copies":19588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":24132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5957},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9945","reported_seq":10019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.152886+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:10.152886+0000","last_peered":"2024-04-22T08:31:10.152886+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:10.152886+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719570,"num_objects":9943,"num_object_clones":0,"num_object_copies":19886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9946","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.095479+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:10.095479+0000","last_peered":"2024-04-22T08:31:10.095479+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:10.095479+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40692132,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6057},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":650940245,"num_objects":158932,"num_object_clones":0,"num_object_copies":317864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":158932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132534,"num_write_kb":530049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":20742761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1320910848,"data_stored":1320830831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159059,"ondisk_log_size":159059,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713244,"kb_used_data":438804,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93658596,"statfs":{"total":96636764160,"available":95906402304,"internally_reserved":0,"allocated":449335296,"data_stored":449123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":2.831},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":2.7519999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.798},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":3.758}]}]},{"osd":1,"up_from":33,"seq":141733920808,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564840,"kb_used_data":312544,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93807000,"statfs":{"total":96636764160,"available":96058368000,"internally_reserved":0,"allocated":320045056,"data_stored":319764066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":258338508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.3410000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.23100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.0760000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.2290000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313274,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":858412,"kb_used_data":538084,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93513428,"statfs":{"total":96636764160,"available":95757750272,"internally_reserved":0,"allocated":550998016,"data_stored":550793068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328003283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":550842368,"data_stored":550838889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320274432,"data_stored":320201373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449794048,"data_stored":449790569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10028", 'reported_seq': 10090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.178304+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:10.178304+0000', 'last_peered': '2024-04-22T08:31:10.178304+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:10.178304+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41074688, 'num_objects': 10028, 'num_object_clones': 0, 'num_object_copies': 20056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10028, 'num_write_kb': 40112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6234}, {'shards': '0,2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9861", 'reported_seq': 9984, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.301599+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:10.301599+0000', 'last_peered': '2024-04-22T08:31:10.301599+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:10.301599+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9861, 'log_dups_size': 0, 'ondisk_log_size': 9861, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40374272, 'num_objects': 9857, 'num_object_clones': 0, 'num_object_copies': 19714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3756, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6688, 'num_write_kb': 26752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3755}, {'shards': '2', 'objects': 6102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:12.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'9962", 'reported_seq': 10124, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.111784+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:10.111784+0000', 'last_peered': '2024-04-22T08:31:10.111784+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:10.111784+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 9962, 'log_dups_size': 0, 'ondisk_log_size': 9962, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40783872, 'num_objects': 9957, 'num_object_clones': 0, 'num_object_copies': 19914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6831, 'num_write_kb': 27324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6078}, {'shards': '0,2', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10082", 'reported_seq': 10205, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.315786+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:10.315786+0000', 'last_peered': '2024-04-22T08:31:10.315786+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:10.315786+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41030866, 'num_objects': 10019, 'num_object_clones': 0, 'num_object_copies': 20038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10022, 'num_write_kb': 40072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3888}, {'shards': '2', 'objects': 6131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10015", 'reported_seq': 10093, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.102091+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:10.102091+0000', 'last_peered': '2024-04-22T08:31:10.102091+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:10.102091+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40989906, 'num_objects': 10009, 'num_object_clones': 0, 'num_object_copies': 20018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3773, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10013, 'num_write_kb': 40032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6238}, {'shards': '0,2', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'9905", 'reported_seq': 10050, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.356342+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:10.356342+0000', 'last_peered': '2024-04-22T08:31:10.356342+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:10.356342+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 9905, 'log_dups_size': 0, 'ondisk_log_size': 9905, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40550400, 'num_objects': 9900, 'num_object_clones': 0, 'num_object_copies': 19800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6359, 'num_write_kb': 25436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3750}, {'shards': '2', 'objects': 6150}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:12.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'9897", 'reported_seq': 12512, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:10.303582+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:10.303582+0000', 'last_peered': '2024-04-22T08:31:10.303582+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:10.303582+0000', 'last_undegraded': '2024-04-22T08:31:10.303582+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40514770, 'num_objects': 9893, 'num_object_clones': 0, 'num_object_copies': 19786, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2524, 'num_objects_unfound': 0, 'num_objects_dirty': 9893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9895, 'num_write_kb': 39567, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1281, 'num_bytes_recovered': 5238784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2524}, {'shards': '2', 'objects': 7369}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10009", 'reported_seq': 10088, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.072111+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:10.072111+0000', 'last_peered': '2024-04-22T08:31:10.072111+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:10.072111+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40989289, 'num_objects': 10008, 'num_object_clones': 0, 'num_object_copies': 20016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10008, 'num_write_kb': 40029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6287}, {'shards': '0,2', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'9802", 'reported_seq': 9977, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.149548+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:10.149548+0000', 'last_peered': '2024-04-22T08:31:10.149548+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:10.149548+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 9802, 'log_dups_size': 0, 'ondisk_log_size': 9802, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40116224, 'num_objects': 9794, 'num_object_clones': 0, 'num_object_copies': 19588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3840, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6033, 'num_write_kb': 24132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5957}, {'shards': '0,2', 'objects': 3837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'9945", 'reported_seq': 10019, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.152886+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:10.152886+0000', 'last_peered': '2024-04-22T08:31:10.152886+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:10.152886+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 9945, 'log_dups_size': 0, 'ondisk_log_size': 9945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40719570, 'num_objects': 9943, 'num_object_clones': 0, 'num_object_copies': 19886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9943, 'num_write_kb': 39766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6238}, {'shards': '0,2', 'objects': 3705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:12.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'9946", 'reported_seq': 10090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:10.095479+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:10.095479+0000', 'last_peered': '2024-04-22T08:31:10.095479+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:10.095479+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 9946, 'log_dups_size': 0, 'ondisk_log_size': 9946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40692132, 'num_objects': 9938, 'num_object_clones': 0, 'num_object_copies': 19876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3884, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9943, 'num_write_kb': 39743, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6057}, {'shards': '0,2', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:12.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:12.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Total time run: 240.018 2024-04-22T08:31:12.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Total writes made: 160037 2024-04-22T08:31:12.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Write size: 4096 2024-04-22T08:31:12.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Object size: 4096 2024-04-22T08:31:12.917 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Bandwidth (MB/sec): 2.60458 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Stddev Bandwidth: 1.753 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Max bandwidth (MB/sec): 11.3477 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Min bandwidth (MB/sec): 0 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Average IOPS: 666 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Stddev IOPS: 448.77 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Max IOPS: 2905 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Min IOPS: 0 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Average Latency(s): 0.0299906 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Stddev Latency(s): 0.0471992 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Max latency(s): 2.74727 2024-04-22T08:31:12.918 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Min latency(s): 0.00109492 2024-04-22T08:31:12.927 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-04-22T08:31:13.700 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:13.700 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37999/317864 objects degraded (11.954%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":308,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":158932,"data_bytes":650940245,"bytes_used":2187771904,"bytes_avail":287722520576,"bytes_total":289910292480,"degraded_objects":37999,"degraded_total":317864,"degraded_ratio":0.11954483678554351,"misplaced_objects":2524,"misplaced_total":317864,"misplaced_ratio":0.007940502856567588,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":113977,"recovering_keys_per_sec":0,"num_objects_recovered":334,"num_bytes_recovered":1368064,"num_keys_recovered":0,"write_bytes_sec":1723649,"read_op_per_sec":0,"write_op_per_sec":420},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:13.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:13.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:14.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:14.446 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:14.549 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-04-22T08:31:14.236744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651251541,"num_objects":159008,"num_object_clones":0,"num_object_copies":318016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":159008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132662,"num_write_kb":530457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":20742761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159187,"ondisk_log_size":159187,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2146668,"kb_used_data":1295508,"kb_used_omap":12,"kb_used_meta":851123,"kb_avail":280968852,"statfs":{"total":289910292480,"available":287712104448,"internally_reserved":0,"allocated":1326600192,"data_stored":1325898772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":871550062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,4,9,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17604608,"num_objects":4298,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-235,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4350,"num_write_kb":17296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":962560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"2.f","version":"37'10028","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.178304+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:10.178304+0000","last_peered":"2024-04-22T08:31:10.178304+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:10.178304+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41074688,"num_objects":10028,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":40112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6234},{"shards":"0,2","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10056","reported_seq":10171,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.317016+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:10.317016+0000","last_peered":"2024-04-22T08:31:10.317016+0000","last_clean":"2024-04-22T08:31:10.317016+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:10.317016+0000","last_undegraded":"2024-04-22T08:31:10.317016+0000","last_fullsized":"2024-04-22T08:31:10.317016+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41164800,"num_objects":10050,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":28448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9861","reported_seq":9984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.301599+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:10.301599+0000","last_peered":"2024-04-22T08:31:10.301599+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:10.301599+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9861,"log_dups_size":0,"ondisk_log_size":9861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40374272,"num_objects":9857,"num_object_clones":0,"num_object_copies":19714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3755},{"shards":"2","objects":6102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'9985","reported_seq":10015,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.352793+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:10.352793+0000","last_peered":"2024-04-22T08:31:10.352793+0000","last_clean":"2024-04-22T08:31:10.352793+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:10.352793+0000","last_undegraded":"2024-04-22T08:31:10.352793+0000","last_fullsized":"2024-04-22T08:31:10.352793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40898560,"num_objects":9985,"num_object_clones":0,"num_object_copies":19970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":39940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'9910","reported_seq":10048,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:10.149502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:10.149502+0000","last_peered":"2024-04-22T08:31:10.149502+0000","last_clean":"2024-04-22T08:31:10.149502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:10.149502+0000","last_undegraded":"2024-04-22T08:31:10.149502+0000","last_fullsized":"2024-04-22T08:31:10.149502+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7002,"num_write_kb":28008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'9962","reported_seq":10124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.111784+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:10.111784+0000","last_peered":"2024-04-22T08:31:10.111784+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:10.111784+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40783872,"num_objects":9957,"num_object_clones":0,"num_object_copies":19914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":27324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6078},{"shards":"0,2","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9961","reported_seq":17697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:13.792451+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:13.792451+0000","last_peered":"2024-04-22T08:31:13.792451+0000","last_clean":"2024-04-22T08:31:13.792451+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:13.792451+0000","last_undegraded":"2024-04-22T08:31:13.792451+0000","last_fullsized":"2024-04-22T08:31:13.792451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9823","reported_seq":9968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:09.989620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:09.989620+0000","last_peered":"2024-04-22T08:31:09.989620+0000","last_clean":"2024-04-22T08:31:09.989620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:09.989620+0000","last_undegraded":"2024-04-22T08:31:09.989620+0000","last_fullsized":"2024-04-22T08:31:09.989620+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40226816,"num_objects":9821,"num_object_clones":0,"num_object_copies":19642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":25884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10082","reported_seq":10205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.315786+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:10.315786+0000","last_peered":"2024-04-22T08:31:10.315786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:10.315786+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41030866,"num_objects":10019,"num_object_clones":0,"num_object_copies":20038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10022,"num_write_kb":40072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3888},{"shards":"2","objects":6131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147590+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:09.147590+0000","last_peered":"2024-04-22T08:30:09.147590+0000","last_clean":"2024-04-22T08:30:09.147590+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:09.147590+0000","last_undegraded":"2024-04-22T08:30:09.147590+0000","last_fullsized":"2024-04-22T08:30:09.147590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10015","reported_seq":10093,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.102091+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:10.102091+0000","last_peered":"2024-04-22T08:31:10.102091+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:10.102091+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989906,"num_objects":10009,"num_object_clones":0,"num_object_copies":20018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":40032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'9905","reported_seq":10050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.356342+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:10.356342+0000","last_peered":"2024-04-22T08:31:10.356342+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:10.356342+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40550400,"num_objects":9900,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":25436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3750},{"shards":"2","objects":6150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'9897","reported_seq":12512,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:10.303582+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:10.303582+0000","last_peered":"2024-04-22T08:31:10.303582+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:10.303582+0000","last_undegraded":"2024-04-22T08:31:10.303582+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40514770,"num_objects":9893,"num_object_clones":0,"num_object_copies":19786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":9893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":39567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1281,"num_bytes_recovered":5238784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2524},{"shards":"2","objects":7369}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10009","reported_seq":10088,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.072111+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:10.072111+0000","last_peered":"2024-04-22T08:31:10.072111+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:10.072111+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989289,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6287},{"shards":"0,2","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9802","reported_seq":9977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.149548+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:10.149548+0000","last_peered":"2024-04-22T08:31:10.149548+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:10.149548+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40116224,"num_objects":9794,"num_object_clones":0,"num_object_copies":19588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":24132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5957},{"shards":"0,2","objects":3837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147683+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:09.147683+0000","last_peered":"2024-04-22T08:30:09.147683+0000","last_clean":"2024-04-22T08:30:09.147683+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:09.147683+0000","last_undegraded":"2024-04-22T08:30:09.147683+0000","last_fullsized":"2024-04-22T08:30:09.147683+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'9945","reported_seq":10019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.152886+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:10.152886+0000","last_peered":"2024-04-22T08:31:10.152886+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:10.152886+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719570,"num_objects":9943,"num_object_clones":0,"num_object_copies":19886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6238},{"shards":"0,2","objects":3705}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'9946","reported_seq":10090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:10.095479+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:10.095479+0000","last_peered":"2024-04-22T08:31:10.095479+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:10.095479+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40692132,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":39743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6057},{"shards":"0,2","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":651251541,"num_objects":159008,"num_object_clones":0,"num_object_copies":318016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37999,"num_objects_misplaced":2524,"num_objects_unfound":0,"num_objects_dirty":159008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132662,"num_write_kb":530457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":20742761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326555136,"data_stored":1326471047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159187,"ondisk_log_size":159187,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713244,"kb_used_data":438804,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93658596,"statfs":{"total":96636764160,"available":95906402304,"internally_reserved":0,"allocated":449335296,"data_stored":449123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.93700000000000006,"15min":0.83499999999999996},"min":{"1min":0.23200000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":7.024,"5min":7.024,"15min":7.024},"last":2.831},{"interface":"front","average":{"1min":1.5189999999999999,"5min":0.85599999999999998,"15min":0.64200000000000002},"min":{"1min":0.182,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.0579999999999998,"5min":7.0579999999999998,"15min":7.0579999999999998},"last":2.7519999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.798},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":3.758}]}]},{"osd":1,"up_from":33,"seq":141733920809,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575012,"kb_used_data":318620,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93796828,"statfs":{"total":96636764160,"available":96047951872,"internally_reserved":0,"allocated":326266880,"data_stored":325981818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":262532812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313274,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":858412,"kb_used_data":538084,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93513428,"statfs":{"total":96636764160,"available":95757750272,"internally_reserved":0,"allocated":550998016,"data_stored":550793068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":328003283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.86099999999999999,"15min":0.747},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.9580000000000002,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.0499999999999998,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":550842368,"data_stored":550838889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325918720,"data_stored":325841589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449794048,"data_stored":449790569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:17.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:18.119 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:18.150 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-04-22T08:31:16.237212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651779949,"num_objects":159138,"num_object_clones":0,"num_object_copies":318276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37968,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":159138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134566,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5243,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160391,"ondisk_log_size":160391,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2158864,"kb_used_data":1297844,"kb_used_omap":12,"kb_used_meta":860979,"kb_avail":280956656,"statfs":{"total":289910292480,"available":287699615744,"internally_reserved":0,"allocated":1328992256,"data_stored":1328286764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":881642606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,4,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17575960,"num_objects":4292,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-31,"num_objects_misplaced":-365,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6118,"num_write_kb":20817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":1495040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"2.f","version":"37'10154","reported_seq":10225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.165356+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:15.165356+0000","last_peered":"2024-04-22T08:31:15.165356+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:15.165356+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41017344,"num_objects":10014,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6224},{"shards":"0,2","objects":3724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10192","reported_seq":10307,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.299661+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:15.299661+0000","last_peered":"2024-04-22T08:31:15.299661+0000","last_clean":"2024-04-22T08:31:15.299661+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:15.299661+0000","last_undegraded":"2024-04-22T08:31:15.299661+0000","last_fullsized":"2024-04-22T08:31:15.299661+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123840,"num_objects":10040,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9985","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.262495+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:15.262495+0000","last_peered":"2024-04-22T08:31:15.262495+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:15.262495+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40460312,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6812,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3704},{"shards":"2","objects":6125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10120","reported_seq":10150,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.296612+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:15.296612+0000","last_peered":"2024-04-22T08:31:15.296612+0000","last_clean":"2024-04-22T08:31:15.296612+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:15.296612+0000","last_undegraded":"2024-04-22T08:31:15.296612+0000","last_fullsized":"2024-04-22T08:31:15.296612+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40935424,"num_objects":9994,"num_object_clones":0,"num_object_copies":19988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10009","reported_seq":10148,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.189871+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:15.189871+0000","last_peered":"2024-04-22T08:31:15.189871+0000","last_clean":"2024-04-22T08:31:15.189871+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:15.189871+0000","last_undegraded":"2024-04-22T08:31:15.189871+0000","last_fullsized":"2024-04-22T08:31:15.189871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40591360,"num_objects":9910,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10081","reported_seq":10259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.136705+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:15.136705+0000","last_peered":"2024-04-22T08:31:15.136705+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:15.136705+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40820736,"num_objects":9966,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6095},{"shards":"0,2","objects":3824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9961","reported_seq":17697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:13.792451+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:13.792451+0000","last_peered":"2024-04-22T08:31:13.792451+0000","last_clean":"2024-04-22T08:31:13.792451+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:13.792451+0000","last_undegraded":"2024-04-22T08:31:13.792451+0000","last_fullsized":"2024-04-22T08:31:13.792451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9961","reported_seq":10106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.067779+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:15.067779+0000","last_peered":"2024-04-22T08:31:15.067779+0000","last_clean":"2024-04-22T08:31:15.067779+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:15.067779+0000","last_undegraded":"2024-04-22T08:31:15.067779+0000","last_fullsized":"2024-04-22T08:31:15.067779+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40325120,"num_objects":9845,"num_object_clones":0,"num_object_copies":19690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10215","reported_seq":10338,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.290369+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:15.290369+0000","last_peered":"2024-04-22T08:31:15.290369+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:15.290369+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40985810,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3816},{"shards":"2","objects":6120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10166","reported_seq":10270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.170210+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:15.170210+0000","last_peered":"2024-04-22T08:31:15.170210+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:15.170210+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41043154,"num_objects":10022,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6264},{"shards":"0,2","objects":3702}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10026","reported_seq":10173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.338852+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:15.338852+0000","last_peered":"2024-04-22T08:31:15.338852+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:15.338852+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3691},{"shards":"2","objects":6153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10020","reported_seq":12895,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:15.363345+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:15.363345+0000","last_peered":"2024-04-22T08:31:15.363345+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:15.363345+0000","last_undegraded":"2024-04-22T08:31:15.363345+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40608978,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":5771264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":7522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10133","reported_seq":10216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.097835+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:15.097835+0000","last_peered":"2024-04-22T08:31:15.097835+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:15.097835+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40997481,"num_objects":10010,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10132,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6291},{"shards":"0,2","objects":3660}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9931","reported_seq":10108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.143550+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:15.143550+0000","last_peered":"2024-04-22T08:31:15.143550+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:15.143550+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40218624,"num_objects":9819,"num_object_clones":0,"num_object_copies":19638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5983},{"shards":"0,2","objects":3785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10076","reported_seq":10153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.055100+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:15.055100+0000","last_peered":"2024-04-22T08:31:15.055100+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:15.055100+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40805586,"num_objects":9964,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6259},{"shards":"0,2","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10061","reported_seq":10209,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.059601+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:15.059601+0000","last_peered":"2024-04-22T08:31:15.059601+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:15.059601+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40720804,"num_objects":9945,"num_object_clones":0,"num_object_copies":19890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6066},{"shards":"0,2","objects":3827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":651779949,"num_objects":159138,"num_object_clones":0,"num_object_copies":318276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37968,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":159138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134566,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5243,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1327169536,"data_stored":1327081375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160391,"ondisk_log_size":160391,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117047,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702604,"kb_used_data":440448,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93669236,"statfs":{"total":96636764160,"available":95917297664,"internally_reserved":0,"allocated":451018752,"data_stored":450803270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920809,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575012,"kb_used_data":318620,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93796828,"statfs":{"total":96636764160,"available":96047951872,"internally_reserved":0,"allocated":326266880,"data_stored":325981818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":262532812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313275,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":881248,"kb_used_data":538776,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93490592,"statfs":{"total":96636764160,"available":95734366208,"internally_reserved":0,"allocated":551706624,"data_stored":551501676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30599999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":551280640,"data_stored":551277161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325918720,"data_stored":325841589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449970176,"data_stored":449962625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:18.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:18.770 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:18.804 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-04-22T08:31:18.237715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651779949,"num_objects":159138,"num_object_clones":0,"num_object_copies":318276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37968,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":159138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134566,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5243,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160391,"ondisk_log_size":160391,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2158864,"kb_used_data":1297844,"kb_used_omap":12,"kb_used_meta":860979,"kb_avail":280956656,"statfs":{"total":289910292480,"available":287699615744,"internally_reserved":0,"allocated":1328992256,"data_stored":1328286764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":881642606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,4,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9199640,"num_objects":2247,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-31,"num_objects_misplaced":-258,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4073,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"2.f","version":"37'10154","reported_seq":10225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.165356+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:15.165356+0000","last_peered":"2024-04-22T08:31:15.165356+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:15.165356+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41017344,"num_objects":10014,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6224},{"shards":"0,2","objects":3724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10192","reported_seq":10307,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.299661+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:15.299661+0000","last_peered":"2024-04-22T08:31:15.299661+0000","last_clean":"2024-04-22T08:31:15.299661+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:15.299661+0000","last_undegraded":"2024-04-22T08:31:15.299661+0000","last_fullsized":"2024-04-22T08:31:15.299661+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123840,"num_objects":10040,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9985","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.262495+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:15.262495+0000","last_peered":"2024-04-22T08:31:15.262495+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:15.262495+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40460312,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6812,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3704},{"shards":"2","objects":6125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10120","reported_seq":10150,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.296612+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:15.296612+0000","last_peered":"2024-04-22T08:31:15.296612+0000","last_clean":"2024-04-22T08:31:15.296612+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:15.296612+0000","last_undegraded":"2024-04-22T08:31:15.296612+0000","last_fullsized":"2024-04-22T08:31:15.296612+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40935424,"num_objects":9994,"num_object_clones":0,"num_object_copies":19988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10009","reported_seq":10148,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.189871+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:15.189871+0000","last_peered":"2024-04-22T08:31:15.189871+0000","last_clean":"2024-04-22T08:31:15.189871+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:15.189871+0000","last_undegraded":"2024-04-22T08:31:15.189871+0000","last_fullsized":"2024-04-22T08:31:15.189871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40591360,"num_objects":9910,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10081","reported_seq":10259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.136705+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:15.136705+0000","last_peered":"2024-04-22T08:31:15.136705+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:15.136705+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40820736,"num_objects":9966,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6095},{"shards":"0,2","objects":3824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'9961","reported_seq":17697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:13.792451+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:13.792451+0000","last_peered":"2024-04-22T08:31:13.792451+0000","last_clean":"2024-04-22T08:31:13.792451+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:13.792451+0000","last_undegraded":"2024-04-22T08:31:13.792451+0000","last_fullsized":"2024-04-22T08:31:13.792451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'9961","reported_seq":10106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.067779+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:15.067779+0000","last_peered":"2024-04-22T08:31:15.067779+0000","last_clean":"2024-04-22T08:31:15.067779+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:15.067779+0000","last_undegraded":"2024-04-22T08:31:15.067779+0000","last_fullsized":"2024-04-22T08:31:15.067779+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":9961,"log_dups_size":0,"ondisk_log_size":9961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40325120,"num_objects":9845,"num_object_clones":0,"num_object_copies":19690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10215","reported_seq":10338,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.290369+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:15.290369+0000","last_peered":"2024-04-22T08:31:15.290369+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:15.290369+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40985810,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3816},{"shards":"2","objects":6120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10166","reported_seq":10270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.170210+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:15.170210+0000","last_peered":"2024-04-22T08:31:15.170210+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:15.170210+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41043154,"num_objects":10022,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6264},{"shards":"0,2","objects":3702}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10026","reported_seq":10173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.338852+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:15.338852+0000","last_peered":"2024-04-22T08:31:15.338852+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:15.338852+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3691},{"shards":"2","objects":6153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10020","reported_seq":12895,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:15.363345+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:15.363345+0000","last_peered":"2024-04-22T08:31:15.363345+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:15.363345+0000","last_undegraded":"2024-04-22T08:31:15.363345+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40608978,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":5771264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":7522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10133","reported_seq":10216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.097835+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:15.097835+0000","last_peered":"2024-04-22T08:31:15.097835+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:15.097835+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40997481,"num_objects":10010,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10132,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6291},{"shards":"0,2","objects":3660}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'9931","reported_seq":10108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.143550+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:15.143550+0000","last_peered":"2024-04-22T08:31:15.143550+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:15.143550+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40218624,"num_objects":9819,"num_object_clones":0,"num_object_copies":19638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5983},{"shards":"0,2","objects":3785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10076","reported_seq":10153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.055100+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:15.055100+0000","last_peered":"2024-04-22T08:31:15.055100+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:15.055100+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40805586,"num_objects":9964,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6259},{"shards":"0,2","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10061","reported_seq":10209,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.059601+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:15.059601+0000","last_peered":"2024-04-22T08:31:15.059601+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:15.059601+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40720804,"num_objects":9945,"num_object_clones":0,"num_object_copies":19890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6066},{"shards":"0,2","objects":3827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":651779949,"num_objects":159138,"num_object_clones":0,"num_object_copies":318276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37968,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":159138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134566,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5243,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1327169536,"data_stored":1327081375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160391,"ondisk_log_size":160391,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117047,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702604,"kb_used_data":440448,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93669236,"statfs":{"total":96636764160,"available":95917297664,"internally_reserved":0,"allocated":451018752,"data_stored":450803270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920809,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575012,"kb_used_data":318620,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93796828,"statfs":{"total":96636764160,"available":96047951872,"internally_reserved":0,"allocated":326266880,"data_stored":325981818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":262532812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313275,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":881248,"kb_used_data":538776,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93490592,"statfs":{"total":96636764160,"available":95734366208,"internally_reserved":0,"allocated":551706624,"data_stored":551501676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30599999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":551280640,"data_stored":551277161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325918720,"data_stored":325841589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449970176,"data_stored":449962625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:18.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:18.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10154", 'reported_seq': 10225, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.165356+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:15.165356+0000', 'last_peered': '2024-04-22T08:31:15.165356+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:15.165356+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41017344, 'num_objects': 10014, 'num_object_clones': 0, 'num_object_copies': 20028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3790, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10154, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6224}, {'shards': '0,2', 'objects': 3724}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:18.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'9985", 'reported_seq': 10111, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.262495+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:15.262495+0000', 'last_peered': '2024-04-22T08:31:15.262495+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:15.262495+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 9985, 'log_dups_size': 0, 'ondisk_log_size': 9985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40460312, 'num_objects': 9879, 'num_object_clones': 0, 'num_object_copies': 19758, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3755, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9879, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6812, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3704}, {'shards': '2', 'objects': 6125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10081", 'reported_seq': 10259, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.136705+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:15.136705+0000', 'last_peered': '2024-04-22T08:31:15.136705+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:15.136705+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40820736, 'num_objects': 9966, 'num_object_clones': 0, 'num_object_copies': 19932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3873, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6950, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6095}, {'shards': '0,2', 'objects': 3824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10215", 'reported_seq': 10338, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.290369+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:15.290369+0000', 'last_peered': '2024-04-22T08:31:15.290369+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:15.290369+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40985810, 'num_objects': 10008, 'num_object_clones': 0, 'num_object_copies': 20016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10155, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3816}, {'shards': '2', 'objects': 6120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10166", 'reported_seq': 10270, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.170210+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:15.170210+0000', 'last_peered': '2024-04-22T08:31:15.170210+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:15.170210+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41043154, 'num_objects': 10022, 'num_object_clones': 0, 'num_object_copies': 20044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10164, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6264}, {'shards': '0,2', 'objects': 3702}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10026", 'reported_seq': 10173, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.338852+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:15.338852+0000', 'last_peered': '2024-04-22T08:31:15.338852+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:15.338852+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40562688, 'num_objects': 9903, 'num_object_clones': 0, 'num_object_copies': 19806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6480, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3691}, {'shards': '2', 'objects': 6153}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:18.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10020", 'reported_seq': 12895, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:15.363345+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:15.363345+0000', 'last_peered': '2024-04-22T08:31:15.363345+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:15.363345+0000', 'last_undegraded': '2024-04-22T08:31:15.363345+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40608978, 'num_objects': 9916, 'num_object_clones': 0, 'num_object_copies': 19832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2394, 'num_objects_unfound': 0, 'num_objects_dirty': 9916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10018, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1411, 'num_bytes_recovered': 5771264, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2394}, {'shards': '2', 'objects': 7522}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10133", 'reported_seq': 10216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.097835+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:15.097835+0000', 'last_peered': '2024-04-22T08:31:15.097835+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:15.097835+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40997481, 'num_objects': 10010, 'num_object_clones': 0, 'num_object_copies': 20020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10132, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6291}, {'shards': '0,2', 'objects': 3660}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'9931", 'reported_seq': 10108, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.143550+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:15.143550+0000', 'last_peered': '2024-04-22T08:31:15.143550+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:15.143550+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 9931, 'log_dups_size': 0, 'ondisk_log_size': 9931, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40218624, 'num_objects': 9819, 'num_object_clones': 0, 'num_object_copies': 19638, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3839, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6162, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5983}, {'shards': '0,2', 'objects': 3785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10076", 'reported_seq': 10153, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.055100+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:15.055100+0000', 'last_peered': '2024-04-22T08:31:15.055100+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:15.055100+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40805586, 'num_objects': 9964, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3707, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10074, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6259}, {'shards': '0,2', 'objects': 3650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:18.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10061", 'reported_seq': 10209, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:15.059601+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:15.059601+0000', 'last_peered': '2024-04-22T08:31:15.059601+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:15.059601+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40720804, 'num_objects': 9945, 'num_object_clones': 0, 'num_object_copies': 19890, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3882, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10058, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6066}, {'shards': '0,2', 'objects': 3827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:18.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:19.718 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:19.718 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37968/318276 objects degraded (11.929%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":159138,"data_bytes":651779949,"bytes_used":2210676736,"bytes_avail":287699615744,"bytes_total":289910292480,"degraded_objects":37968,"degraded_total":318276,"degraded_ratio":0.11929268936394827,"misplaced_objects":2394,"misplaced_total":318276,"misplaced_ratio":0.007521773555027712,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":88042,"recovering_keys_per_sec":0,"num_objects_recovered":289,"num_bytes_recovered":1056768,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1078096,"read_op_per_sec":0,"write_op_per_sec":339},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:19.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:19.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:20.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:20.474 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-04-22T08:31:20.238160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646668141,"num_objects":157890,"num_object_clones":0,"num_object_copies":315780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37897,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":157890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135814,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5314,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2148172,"kb_used_data":1291244,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280967348,"statfs":{"total":289910292480,"available":287710564352,"internally_reserved":0,"allocated":1322233856,"data_stored":1321528364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":877448302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,4,5,13,2,0,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4087832,"num_objects":999,"num_object_clones":0,"num_object_copies":1998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":-258,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5321,"num_write_kb":12637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002831"},"pg_stats":[{"pgid":"2.f","version":"37'10270","reported_seq":10367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.179662+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:20.179662+0000","last_peered":"2024-04-22T08:31:20.179662+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:20.179662+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40542208,"num_objects":9898,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6121},{"shards":"0,2","objects":3608}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10192","reported_seq":10307,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.299661+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:15.299661+0000","last_peered":"2024-04-22T08:31:15.299661+0000","last_clean":"2024-04-22T08:31:15.299661+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:15.299661+0000","last_undegraded":"2024-04-22T08:31:15.299661+0000","last_fullsized":"2024-04-22T08:31:15.299661+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123840,"num_objects":10040,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'9985","reported_seq":10111,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.262495+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:15.262495+0000","last_peered":"2024-04-22T08:31:15.262495+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:15.262495+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40460312,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6812,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3704},{"shards":"2","objects":6125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10120","reported_seq":10150,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:15.296612+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:15.296612+0000","last_peered":"2024-04-22T08:31:15.296612+0000","last_clean":"2024-04-22T08:31:15.296612+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:15.296612+0000","last_undegraded":"2024-04-22T08:31:15.296612+0000","last_fullsized":"2024-04-22T08:31:15.296612+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40935424,"num_objects":9994,"num_object_clones":0,"num_object_copies":19988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10124","reported_seq":10263,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.179490+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:20.179490+0000","last_peered":"2024-04-22T08:31:20.179490+0000","last_clean":"2024-04-22T08:31:20.179490+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:20.179490+0000","last_undegraded":"2024-04-22T08:31:20.179490+0000","last_fullsized":"2024-04-22T08:31:20.179490+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40120320,"num_objects":9795,"num_object_clones":0,"num_object_copies":19590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10203","reported_seq":10404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.127590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:20.127590+0000","last_peered":"2024-04-22T08:31:20.127590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:20.127590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40321024,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7072,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10088","reported_seq":17824,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:18.647819+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:18.647819+0000","last_peered":"2024-04-22T08:31:18.647819+0000","last_clean":"2024-04-22T08:31:18.647819+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:18.647819+0000","last_undegraded":"2024-04-22T08:31:18.647819+0000","last_fullsized":"2024-04-22T08:31:18.647819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":9776,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10090","reported_seq":10235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.134751+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:20.134751+0000","last_peered":"2024-04-22T08:31:20.134751+0000","last_clean":"2024-04-22T08:31:20.134751+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:20.134751+0000","last_undegraded":"2024-04-22T08:31:20.134751+0000","last_fullsized":"2024-04-22T08:31:20.134751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":9716,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10215","reported_seq":10338,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.290369+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:15.290369+0000","last_peered":"2024-04-22T08:31:15.290369+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:15.290369+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40985810,"num_objects":10008,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3816},{"shards":"2","objects":6120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10269","reported_seq":10387,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.013787+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:20.013787+0000","last_peered":"2024-04-22T08:31:20.013787+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:20.013787+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621266,"num_objects":9919,"num_object_clones":0,"num_object_copies":19838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6168},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10026","reported_seq":10173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:15.338852+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:15.338852+0000","last_peered":"2024-04-22T08:31:15.338852+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:15.338852+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3691},{"shards":"2","objects":6153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10020","reported_seq":12895,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:15.363345+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:15.363345+0000","last_peered":"2024-04-22T08:31:15.363345+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:15.363345+0000","last_undegraded":"2024-04-22T08:31:15.363345+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40608978,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":5771264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":7522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10259","reported_seq":10362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.197948+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:20.197948+0000","last_peered":"2024-04-22T08:31:20.197948+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:20.197948+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481385,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6175},{"shards":"0,2","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10063","reported_seq":10261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.212599+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:20.212599+0000","last_peered":"2024-04-22T08:31:20.212599+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:20.212599+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39677952,"num_objects":9687,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5862},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10215","reported_seq":10303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.184000+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:20.184000+0000","last_peered":"2024-04-22T08:31:20.184000+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:20.184000+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236242,"num_objects":9825,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6126},{"shards":"0,2","objects":3511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10200","reported_seq":10372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.166525+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:20.166525+0000","last_peered":"2024-04-22T08:31:20.166525+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:20.166525+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40151460,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5939},{"shards":"0,2","objects":3688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":646668141,"num_objects":157890,"num_object_clones":0,"num_object_copies":315780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37897,"num_objects_misplaced":2394,"num_objects_unfound":0,"num_objects_dirty":157890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135814,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5314,"num_bytes_recovered":21275241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1320488960,"data_stored":1320400799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117047,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702604,"kb_used_data":440448,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93669236,"statfs":{"total":96636764160,"available":95917297664,"internally_reserved":0,"allocated":451018752,"data_stored":450803270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920810,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566296,"kb_used_data":318096,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93805544,"statfs":{"total":96636764160,"available":96056877056,"internally_reserved":0,"allocated":325730304,"data_stored":325445242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":254144204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.4359999999999999},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.516}]}]},{"osd":0,"up_from":21,"seq":90194313276,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879272,"kb_used_data":532700,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93492568,"statfs":{"total":96636764160,"available":95736389632,"internally_reserved":0,"allocated":545484800,"data_stored":545279852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,0,0,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.253},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":545120256,"data_stored":545116777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325398528,"data_stored":325321397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449970176,"data_stored":449962625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:23.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:24.191 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:24.192 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-04-22T08:31:22.238691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643669869,"num_objects":157158,"num_object_clones":0,"num_object_copies":314316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37882,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":157158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136546,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5464,"num_bytes_recovered":21828201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2147628,"kb_used_data":1286608,"kb_used_omap":12,"kb_used_meta":860979,"kb_avail":280967892,"statfs":{"total":289910292480,"available":287711121408,"internally_reserved":0,"allocated":1317486592,"data_stored":1316781100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":881642606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,12,2,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4247528,"num_objects":-1036,"num_object_clones":0,"num_object_copies":-2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":-393,"num_objects_unfound":0,"num_objects_dirty":-1036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4750,"num_write_kb":7425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":1609728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002916"},"pg_stats":[{"pgid":"2.f","version":"37'10270","reported_seq":10367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.179662+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:20.179662+0000","last_peered":"2024-04-22T08:31:20.179662+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:20.179662+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40542208,"num_objects":9898,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6121},{"shards":"0,2","objects":3608}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10326","reported_seq":10441,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.166811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:20.166811+0000","last_peered":"2024-04-22T08:31:20.166811+0000","last_clean":"2024-04-22T08:31:20.166811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:20.166811+0000","last_undegraded":"2024-04-22T08:31:20.166811+0000","last_fullsized":"2024-04-22T08:31:20.166811+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40574976,"num_objects":9906,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10090","reported_seq":10230,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.300350+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:20.300350+0000","last_peered":"2024-04-22T08:31:20.300350+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:20.300350+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40030232,"num_objects":9774,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6917,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3599},{"shards":"2","objects":6027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10238","reported_seq":10268,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.358055+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:20.358055+0000","last_peered":"2024-04-22T08:31:20.358055+0000","last_clean":"2024-04-22T08:31:20.358055+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:20.358055+0000","last_undegraded":"2024-04-22T08:31:20.358055+0000","last_fullsized":"2024-04-22T08:31:20.358055+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40452096,"num_objects":9876,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10124","reported_seq":10263,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.179490+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:20.179490+0000","last_peered":"2024-04-22T08:31:20.179490+0000","last_clean":"2024-04-22T08:31:20.179490+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:20.179490+0000","last_undegraded":"2024-04-22T08:31:20.179490+0000","last_fullsized":"2024-04-22T08:31:20.179490+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40120320,"num_objects":9795,"num_object_clones":0,"num_object_copies":19590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10203","reported_seq":10404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.127590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:20.127590+0000","last_peered":"2024-04-22T08:31:20.127590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:20.127590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40321024,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7072,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10088","reported_seq":17824,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:18.647819+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:18.647819+0000","last_peered":"2024-04-22T08:31:18.647819+0000","last_clean":"2024-04-22T08:31:18.647819+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:18.647819+0000","last_undegraded":"2024-04-22T08:31:18.647819+0000","last_fullsized":"2024-04-22T08:31:18.647819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":9776,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10090","reported_seq":10235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.134751+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:20.134751+0000","last_peered":"2024-04-22T08:31:20.134751+0000","last_clean":"2024-04-22T08:31:20.134751+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:20.134751+0000","last_undegraded":"2024-04-22T08:31:20.134751+0000","last_fullsized":"2024-04-22T08:31:20.134751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":9716,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10320","reported_seq":10451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.175503+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:20.175503+0000","last_peered":"2024-04-22T08:31:20.175503+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:20.175503+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555730,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3711},{"shards":"2","objects":6019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10269","reported_seq":10387,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.013787+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:20.013787+0000","last_peered":"2024-04-22T08:31:20.013787+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:20.013787+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621266,"num_objects":9919,"num_object_clones":0,"num_object_copies":19838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6168},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10160","reported_seq":10315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.323636+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:20.323636+0000","last_peered":"2024-04-22T08:31:20.323636+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:20.323636+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40013824,"num_objects":9769,"num_object_clones":0,"num_object_copies":19538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6614,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3557},{"shards":"2","objects":6023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10156","reported_seq":13301,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:20.366064+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:20.366064+0000","last_peered":"2024-04-22T08:31:20.366064+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:20.366064+0000","last_undegraded":"2024-04-22T08:31:20.366064+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40051922,"num_objects":9780,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":9780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":6324224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2259},{"shards":"2","objects":7521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10259","reported_seq":10362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.197948+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:20.197948+0000","last_peered":"2024-04-22T08:31:20.197948+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:20.197948+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481385,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6175},{"shards":"0,2","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10063","reported_seq":10261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.212599+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:20.212599+0000","last_peered":"2024-04-22T08:31:20.212599+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:20.212599+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39677952,"num_objects":9687,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5862},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10215","reported_seq":10303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.184000+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:20.184000+0000","last_peered":"2024-04-22T08:31:20.184000+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:20.184000+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236242,"num_objects":9825,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6126},{"shards":"0,2","objects":3511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10200","reported_seq":10372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.166525+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:20.166525+0000","last_peered":"2024-04-22T08:31:20.166525+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:20.166525+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40151460,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5939},{"shards":"0,2","objects":3688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":643669869,"num_objects":157158,"num_object_clones":0,"num_object_copies":314316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37882,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":157158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136546,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5464,"num_bytes_recovered":21828201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316458496,"data_stored":1316370335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702060,"kb_used_data":435812,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93669780,"statfs":{"total":96636764160,"available":95917854720,"internally_reserved":0,"allocated":446271488,"data_stored":446056006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.307},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920810,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566296,"kb_used_data":318096,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93805544,"statfs":{"total":96636764160,"available":96056877056,"internally_reserved":0,"allocated":325730304,"data_stored":325445242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":254144204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.4359999999999999},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.516}]}]},{"osd":0,"up_from":21,"seq":90194313276,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879272,"kb_used_data":532700,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93492568,"statfs":{"total":96636764160,"available":95736389632,"internally_reserved":0,"allocated":545484800,"data_stored":545279852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,0,0,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.253},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":545120256,"data_stored":545116777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325398528,"data_stored":325321397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445939712,"data_stored":445932161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:24.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:24.993 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:25.101 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-04-22T08:31:24.239140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643108717,"num_objects":157021,"num_object_clones":0,"num_object_copies":314042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37882,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":157021,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136683,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5464,"num_bytes_recovered":21828201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160808,"ondisk_log_size":160808,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2169348,"kb_used_data":1286184,"kb_used_omap":12,"kb_used_meta":883123,"kb_avail":280946172,"statfs":{"total":289910292480,"available":287688880128,"internally_reserved":0,"allocated":1317052416,"data_stored":1316346924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":904318062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,6,7,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7831528,"num_objects":-1911,"num_object_clones":0,"num_object_copies":-3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":-265,"num_objects_unfound":0,"num_objects_dirty":-1911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4149,"num_write_kb":4473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":1085440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002862"},"pg_stats":[{"pgid":"2.f","version":"37'10270","reported_seq":10367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.179662+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:20.179662+0000","last_peered":"2024-04-22T08:31:20.179662+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:20.179662+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40542208,"num_objects":9898,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6121},{"shards":"0,2","objects":3608}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10326","reported_seq":10441,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.166811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:20.166811+0000","last_peered":"2024-04-22T08:31:20.166811+0000","last_clean":"2024-04-22T08:31:20.166811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:20.166811+0000","last_undegraded":"2024-04-22T08:31:20.166811+0000","last_fullsized":"2024-04-22T08:31:20.166811+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40574976,"num_objects":9906,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10090","reported_seq":10230,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.300350+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:20.300350+0000","last_peered":"2024-04-22T08:31:20.300350+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:20.300350+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40030232,"num_objects":9774,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6917,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3599},{"shards":"2","objects":6027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10238","reported_seq":10268,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.358055+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:20.358055+0000","last_peered":"2024-04-22T08:31:20.358055+0000","last_clean":"2024-04-22T08:31:20.358055+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:20.358055+0000","last_undegraded":"2024-04-22T08:31:20.358055+0000","last_fullsized":"2024-04-22T08:31:20.358055+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40452096,"num_objects":9876,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10124","reported_seq":10263,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.179490+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:20.179490+0000","last_peered":"2024-04-22T08:31:20.179490+0000","last_clean":"2024-04-22T08:31:20.179490+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:20.179490+0000","last_undegraded":"2024-04-22T08:31:20.179490+0000","last_fullsized":"2024-04-22T08:31:20.179490+0000","mapping_epoch":27,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40120320,"num_objects":9795,"num_object_clones":0,"num_object_copies":19590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10203","reported_seq":10404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.127590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:20.127590+0000","last_peered":"2024-04-22T08:31:20.127590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:20.127590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40321024,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7072,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5985},{"shards":"0,2","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10225","reported_seq":17961,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:23.738346+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:23.738346+0000","last_peered":"2024-04-22T08:31:23.738346+0000","last_clean":"2024-04-22T08:31:23.738346+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:23.738346+0000","last_undegraded":"2024-04-22T08:31:23.738346+0000","last_fullsized":"2024-04-22T08:31:23.738346+0000","mapping_epoch":36,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39481344,"num_objects":9639,"num_object_clones":0,"num_object_copies":19278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6593,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10090","reported_seq":10235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:20.134751+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:20.134751+0000","last_peered":"2024-04-22T08:31:20.134751+0000","last_clean":"2024-04-22T08:31:20.134751+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:20.134751+0000","last_undegraded":"2024-04-22T08:31:20.134751+0000","last_fullsized":"2024-04-22T08:31:20.134751+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39796736,"num_objects":9716,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10320","reported_seq":10451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.175503+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:20.175503+0000","last_peered":"2024-04-22T08:31:20.175503+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:20.175503+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555730,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3711},{"shards":"2","objects":6019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10269","reported_seq":10387,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.013787+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:20.013787+0000","last_peered":"2024-04-22T08:31:20.013787+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:20.013787+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621266,"num_objects":9919,"num_object_clones":0,"num_object_copies":19838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6168},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10160","reported_seq":10315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.323636+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:20.323636+0000","last_peered":"2024-04-22T08:31:20.323636+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:20.323636+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40013824,"num_objects":9769,"num_object_clones":0,"num_object_copies":19538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6614,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3557},{"shards":"2","objects":6023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10156","reported_seq":13301,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:20.366064+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:20.366064+0000","last_peered":"2024-04-22T08:31:20.366064+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:20.366064+0000","last_undegraded":"2024-04-22T08:31:20.366064+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40051922,"num_objects":9780,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":9780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":6324224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2259},{"shards":"2","objects":7521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10259","reported_seq":10362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.197948+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:20.197948+0000","last_peered":"2024-04-22T08:31:20.197948+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:20.197948+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40481385,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6175},{"shards":"0,2","objects":3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10063","reported_seq":10261,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.212599+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:20.212599+0000","last_peered":"2024-04-22T08:31:20.212599+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:20.212599+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39677952,"num_objects":9687,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5862},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10215","reported_seq":10303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.184000+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:20.184000+0000","last_peered":"2024-04-22T08:31:20.184000+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:20.184000+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236242,"num_objects":9825,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6126},{"shards":"0,2","objects":3511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10200","reported_seq":10372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:20.166525+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:20.166525+0000","last_peered":"2024-04-22T08:31:20.166525+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:20.166525+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40151460,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5939},{"shards":"0,2","objects":3688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":643108717,"num_objects":157021,"num_object_clones":0,"num_object_copies":314042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37882,"num_objects_misplaced":2259,"num_objects_unfound":0,"num_objects_dirty":157021,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136683,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5464,"num_bytes_recovered":21828201,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316044800,"data_stored":1315956639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160808,"ondisk_log_size":160808,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117048,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702060,"kb_used_data":435812,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93669780,"statfs":{"total":96636764160,"available":95917854720,"internally_reserved":0,"allocated":446271488,"data_stored":446056006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.307},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920811,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588016,"kb_used_data":317672,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93783824,"statfs":{"total":96636764160,"available":96034635776,"internally_reserved":0,"allocated":325296128,"data_stored":325011066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":276819660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.315}]}]},{"osd":0,"up_from":21,"seq":90194313276,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879272,"kb_used_data":532700,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93492568,"statfs":{"total":96636764160,"available":95736389632,"internally_reserved":0,"allocated":545484800,"data_stored":545279852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,0,0,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.253},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.94999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":545120256,"data_stored":545116777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324984832,"data_stored":324907701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445939712,"data_stored":445932161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10270", 'reported_seq': 10367, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.179662+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:20.179662+0000', 'last_peered': '2024-04-22T08:31:20.179662+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:20.179662+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40542208, 'num_objects': 9898, 'num_object_clones': 0, 'num_object_copies': 19796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3777, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10270, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6121}, {'shards': '0,2', 'objects': 3608}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10090", 'reported_seq': 10230, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.300350+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:20.300350+0000', 'last_peered': '2024-04-22T08:31:20.300350+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:20.300350+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40030232, 'num_objects': 9774, 'num_object_clones': 0, 'num_object_copies': 19548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3748, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9774, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6917, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3599}, {'shards': '2', 'objects': 6027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10203", 'reported_seq': 10404, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.127590+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:20.127590+0000', 'last_peered': '2024-04-22T08:31:20.127590+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:20.127590+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40321024, 'num_objects': 9844, 'num_object_clones': 0, 'num_object_copies': 19688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3861, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7072, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5985}, {'shards': '0,2', 'objects': 3702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:25.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10320", 'reported_seq': 10451, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.175503+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:20.175503+0000', 'last_peered': '2024-04-22T08:31:20.175503+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:20.175503+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40555730, 'num_objects': 9903, 'num_object_clones': 0, 'num_object_copies': 19806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3886, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10260, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3711}, {'shards': '2', 'objects': 6019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10269", 'reported_seq': 10387, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.013787+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:20.013787+0000', 'last_peered': '2024-04-22T08:31:20.013787+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:20.013787+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40621266, 'num_objects': 9919, 'num_object_clones': 0, 'num_object_copies': 19838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3753, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10267, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6168}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10160", 'reported_seq': 10315, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.323636+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:20.323636+0000', 'last_peered': '2024-04-22T08:31:20.323636+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:20.323636+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40013824, 'num_objects': 9769, 'num_object_clones': 0, 'num_object_copies': 19538, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3748, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6614, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3557}, {'shards': '2', 'objects': 6023}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10156", 'reported_seq': 13301, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:20.366064+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:20.366064+0000', 'last_peered': '2024-04-22T08:31:20.366064+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:20.366064+0000', 'last_undegraded': '2024-04-22T08:31:20.366064+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40051922, 'num_objects': 9780, 'num_object_clones': 0, 'num_object_copies': 19560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2259, 'num_objects_unfound': 0, 'num_objects_dirty': 9780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10154, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1546, 'num_bytes_recovered': 6324224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2259}, {'shards': '2', 'objects': 7521}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10259", 'reported_seq': 10362, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.197948+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:20.197948+0000', 'last_peered': '2024-04-22T08:31:20.197948+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:20.197948+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40481385, 'num_objects': 9884, 'num_object_clones': 0, 'num_object_copies': 19768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3710, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10258, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6175}, {'shards': '0,2', 'objects': 3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'10063", 'reported_seq': 10261, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.212599+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:20.212599+0000', 'last_peered': '2024-04-22T08:31:20.212599+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:20.212599+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39677952, 'num_objects': 9687, 'num_object_clones': 0, 'num_object_copies': 19374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3828, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6294, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5862}, {'shards': '0,2', 'objects': 3653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:25.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10215", 'reported_seq': 10303, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.184000+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:20.184000+0000', 'last_peered': '2024-04-22T08:31:20.184000+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:20.184000+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40236242, 'num_objects': 9825, 'num_object_clones': 0, 'num_object_copies': 19650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3701, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10213, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6126}, {'shards': '0,2', 'objects': 3511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.105 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:25.105 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10200", 'reported_seq': 10372, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:20.166525+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:20.166525+0000', 'last_peered': '2024-04-22T08:31:20.166525+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:20.166525+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40151460, 'num_objects': 9806, 'num_object_clones': 0, 'num_object_copies': 19612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3870, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10197, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5939}, {'shards': '0,2', 'objects': 3688}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:25.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:26.036 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:26.036 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37882/314042 objects degraded (12.063%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":320,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":157021,"data_bytes":643108717,"bytes_used":2221412352,"bytes_avail":287688880128,"bytes_total":289910292480,"degraded_objects":37882,"degraded_total":314042,"degraded_ratio":0.1206271772565453,"misplaced_objects":2259,"misplaced_total":314042,"misplaced_ratio":0.0071933053540609217,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":90431,"recovering_keys_per_sec":0,"num_objects_recovered":382,"num_bytes_recovered":1085440,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":381604,"read_op_per_sec":0,"write_op_per_sec":345},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:26.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:26.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:26.769 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:26.825 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-04-22T08:31:26.239627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635363181,"num_objects":155130,"num_object_clones":0,"num_object_copies":310260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138574,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160899,"ondisk_log_size":160899,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2155168,"kb_used_data":1276104,"kb_used_omap":12,"kb_used_meta":879027,"kb_avail":280960352,"statfs":{"total":289910292480,"available":287703400448,"internally_reserved":0,"allocated":1306730496,"data_stored":1306025004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":900123758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,4,10,6,0,3,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15888360,"num_objects":-3878,"num_object_clones":0,"num_object_copies":-7756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":-426,"num_objects_unfound":0,"num_objects_dirty":-3878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5912,"num_write_kb":4065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"2.f","version":"37'10378","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.168321+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:25.168321+0000","last_peered":"2024-04-22T08:31:25.168321+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:25.168321+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40099840,"num_objects":9790,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6030},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10454","reported_seq":10569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.305659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:25.305659+0000","last_peered":"2024-04-22T08:31:25.305659+0000","last_clean":"2024-04-22T08:31:25.305659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:25.305659+0000","last_undegraded":"2024-04-22T08:31:25.305659+0000","last_fullsized":"2024-04-22T08:31:25.305659+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":9778,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10229","reported_seq":10381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.363217+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:25.363217+0000","last_peered":"2024-04-22T08:31:25.363217+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:25.363217+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460888,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7056,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3460},{"shards":"2","objects":5894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10356","reported_seq":10386,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.354457+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:25.354457+0000","last_peered":"2024-04-22T08:31:25.354457+0000","last_clean":"2024-04-22T08:31:25.354457+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:25.354457+0000","last_undegraded":"2024-04-22T08:31:25.354457+0000","last_fullsized":"2024-04-22T08:31:25.354457+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":9758,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10267","reported_seq":10406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.199860+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:25.199860+0000","last_peered":"2024-04-22T08:31:25.199860+0000","last_clean":"2024-04-22T08:31:25.199860+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:25.199860+0000","last_undegraded":"2024-04-22T08:31:25.199860+0000","last_fullsized":"2024-04-22T08:31:25.199860+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39534592,"num_objects":9652,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10318","reported_seq":10550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.158523+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:25.158523+0000","last_peered":"2024-04-22T08:31:25.158523+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:25.158523+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39849984,"num_objects":9729,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7187,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5885},{"shards":"0,2","objects":3587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10225","reported_seq":17961,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:23.738346+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:23.738346+0000","last_peered":"2024-04-22T08:31:23.738346+0000","last_clean":"2024-04-22T08:31:23.738346+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:23.738346+0000","last_undegraded":"2024-04-22T08:31:23.738346+0000","last_fullsized":"2024-04-22T08:31:23.738346+0000","mapping_epoch":36,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39481344,"num_objects":9639,"num_object_clones":0,"num_object_copies":19278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6593,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10219","reported_seq":10364,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.222255+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:25.222255+0000","last_peered":"2024-04-22T08:31:25.222255+0000","last_clean":"2024-04-22T08:31:25.222255+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:25.222255+0000","last_undegraded":"2024-04-22T08:31:25.222255+0000","last_fullsized":"2024-04-22T08:31:25.222255+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10461","reported_seq":10616,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.382201+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:25.382201+0000","last_peered":"2024-04-22T08:31:25.382201+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:25.382201+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39978194,"num_objects":9762,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3570},{"shards":"2","objects":5890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10382","reported_seq":10518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.129634+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:25.129634+0000","last_peered":"2024-04-22T08:31:25.129634+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:25.129634+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40158418,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6064},{"shards":"0,2","objects":3486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10279","reported_seq":10448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.288380+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:25.288380+0000","last_peered":"2024-04-22T08:31:25.288380+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:25.288380+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39526400,"num_objects":9650,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3438},{"shards":"2","objects":5911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10293","reported_seq":13760,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:25.402601+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:25.402601+0000","last_peered":"2024-04-22T08:31:25.402601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:25.402601+0000","last_undegraded":"2024-04-22T08:31:25.402601+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39490770,"num_objects":9643,"num_object_clones":0,"num_object_copies":19286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":9643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":6983680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2098},{"shards":"2","objects":7545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10372","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.120299+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:25.120299+0000","last_peered":"2024-04-22T08:31:25.120299+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:25.120299+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40018537,"num_objects":9771,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10196","reported_seq":10419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.178319+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:25.178319+0000","last_peered":"2024-04-22T08:31:25.178319+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:25.178319+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5741},{"shards":"0,2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10343","reported_seq":10464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.187983+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:25.187983+0000","last_peered":"2024-04-22T08:31:25.187983+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:25.187983+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39711954,"num_objects":9697,"num_object_clones":0,"num_object_copies":19394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6014},{"shards":"0,2","objects":3383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10327","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.079072+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:25.079072+0000","last_peered":"2024-04-22T08:31:25.079072+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:25.079072+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39631268,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5826},{"shards":"0,2","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":635363181,"num_objects":155130,"num_object_clones":0,"num_object_copies":310260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138574,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305661440,"data_stored":1305573279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160899,"ondisk_log_size":160899,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117049,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702004,"kb_used_data":431660,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93669836,"statfs":{"total":96636764160,"available":95917912064,"internally_reserved":0,"allocated":442019840,"data_stored":441804358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,0,0,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":4.3609999999999998},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.252}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":4.4050000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920811,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588016,"kb_used_data":317672,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93783824,"statfs":{"total":96636764160,"available":96034635776,"internally_reserved":0,"allocated":325296128,"data_stored":325011066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":276819660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.315}]}]},{"osd":0,"up_from":21,"seq":90194313277,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865148,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93506692,"statfs":{"total":96636764160,"available":95750852608,"internally_reserved":0,"allocated":539414528,"data_stored":539209580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,6,6,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.2610000000000001},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.2999999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539037696,"data_stored":539034217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324984832,"data_stored":324907701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":441638912,"data_stored":441631361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:29.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:30.670 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:30.734 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-04-22T08:31:30.240680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634933101,"num_objects":155025,"num_object_clones":0,"num_object_copies":310050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155025,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138679,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2164680,"kb_used_data":1275760,"kb_used_omap":12,"kb_used_meta":888883,"kb_avail":280950840,"statfs":{"total":289910292480,"available":287693660160,"internally_reserved":0,"allocated":1306378240,"data_stored":1305672748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":910216302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,7,7,8,2,3,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16846848,"num_objects":-4113,"num_object_clones":0,"num_object_copies":-8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":-296,"num_objects_unfound":0,"num_objects_dirty":-4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002964"},"pg_stats":[{"pgid":"2.f","version":"37'10378","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.168321+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:25.168321+0000","last_peered":"2024-04-22T08:31:25.168321+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:25.168321+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40099840,"num_objects":9790,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6030},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10454","reported_seq":10569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.305659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:25.305659+0000","last_peered":"2024-04-22T08:31:25.305659+0000","last_clean":"2024-04-22T08:31:25.305659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:25.305659+0000","last_undegraded":"2024-04-22T08:31:25.305659+0000","last_fullsized":"2024-04-22T08:31:25.305659+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":9778,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10229","reported_seq":10381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.363217+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:25.363217+0000","last_peered":"2024-04-22T08:31:25.363217+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:25.363217+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460888,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7056,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3460},{"shards":"2","objects":5894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10356","reported_seq":10386,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.354457+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:25.354457+0000","last_peered":"2024-04-22T08:31:25.354457+0000","last_clean":"2024-04-22T08:31:25.354457+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:25.354457+0000","last_undegraded":"2024-04-22T08:31:25.354457+0000","last_fullsized":"2024-04-22T08:31:25.354457+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":9758,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10267","reported_seq":10406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.199860+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:25.199860+0000","last_peered":"2024-04-22T08:31:25.199860+0000","last_clean":"2024-04-22T08:31:25.199860+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:25.199860+0000","last_undegraded":"2024-04-22T08:31:25.199860+0000","last_fullsized":"2024-04-22T08:31:25.199860+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39534592,"num_objects":9652,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10318","reported_seq":10550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.158523+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:25.158523+0000","last_peered":"2024-04-22T08:31:25.158523+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:25.158523+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39849984,"num_objects":9729,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7187,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5885},{"shards":"0,2","objects":3587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10330","reported_seq":18066,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:28.861509+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:28.861509+0000","last_peered":"2024-04-22T08:31:28.861509+0000","last_clean":"2024-04-22T08:31:28.861509+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:28.861509+0000","last_undegraded":"2024-04-22T08:31:28.861509+0000","last_fullsized":"2024-04-22T08:31:28.861509+0000","mapping_epoch":36,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39051264,"num_objects":9534,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10219","reported_seq":10364,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.222255+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:25.222255+0000","last_peered":"2024-04-22T08:31:25.222255+0000","last_clean":"2024-04-22T08:31:25.222255+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:25.222255+0000","last_undegraded":"2024-04-22T08:31:25.222255+0000","last_fullsized":"2024-04-22T08:31:25.222255+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10461","reported_seq":10616,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.382201+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:25.382201+0000","last_peered":"2024-04-22T08:31:25.382201+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:25.382201+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39978194,"num_objects":9762,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3570},{"shards":"2","objects":5890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10382","reported_seq":10518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.129634+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:25.129634+0000","last_peered":"2024-04-22T08:31:25.129634+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:25.129634+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40158418,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6064},{"shards":"0,2","objects":3486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10279","reported_seq":10448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.288380+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:25.288380+0000","last_peered":"2024-04-22T08:31:25.288380+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:25.288380+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39526400,"num_objects":9650,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3438},{"shards":"2","objects":5911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10293","reported_seq":13760,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:25.402601+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:25.402601+0000","last_peered":"2024-04-22T08:31:25.402601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:25.402601+0000","last_undegraded":"2024-04-22T08:31:25.402601+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39490770,"num_objects":9643,"num_object_clones":0,"num_object_copies":19286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":9643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":6983680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2098},{"shards":"2","objects":7545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10372","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.120299+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:25.120299+0000","last_peered":"2024-04-22T08:31:25.120299+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:25.120299+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40018537,"num_objects":9771,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10196","reported_seq":10419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.178319+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:25.178319+0000","last_peered":"2024-04-22T08:31:25.178319+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:25.178319+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5741},{"shards":"0,2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10343","reported_seq":10464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.187983+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:25.187983+0000","last_peered":"2024-04-22T08:31:25.187983+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:25.187983+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39711954,"num_objects":9697,"num_object_clones":0,"num_object_copies":19394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6014},{"shards":"0,2","objects":3383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10327","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.079072+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:25.079072+0000","last_peered":"2024-04-22T08:31:25.079072+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:25.079072+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39631268,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5826},{"shards":"0,2","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634933101,"num_objects":155025,"num_object_clones":0,"num_object_copies":310050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155025,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138679,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305251840,"data_stored":1305163679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117049,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702004,"kb_used_data":431660,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93669836,"statfs":{"total":96636764160,"available":95917912064,"internally_reserved":0,"allocated":442019840,"data_stored":441804358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,0,0,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":4.3609999999999998},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.252}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":4.4050000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920812,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597528,"kb_used_data":317328,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93774312,"statfs":{"total":96636764160,"available":96024895488,"internally_reserved":0,"allocated":324943872,"data_stored":324658810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34499999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313277,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865148,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93506692,"statfs":{"total":96636764160,"available":95750852608,"internally_reserved":0,"allocated":539414528,"data_stored":539209580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,6,6,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.2610000000000001},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.2999999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539037696,"data_stored":539034217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324575232,"data_stored":324498101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":441638912,"data_stored":441631361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:30.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:31.501 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:31.502 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:31.569 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-04-22T08:31:30.240680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634933101,"num_objects":155025,"num_object_clones":0,"num_object_copies":310050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155025,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138679,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2164680,"kb_used_data":1275760,"kb_used_omap":12,"kb_used_meta":888883,"kb_avail":280950840,"statfs":{"total":289910292480,"available":287693660160,"internally_reserved":0,"allocated":1306378240,"data_stored":1305672748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":910216302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,7,7,8,2,3,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16846848,"num_objects":-4113,"num_object_clones":0,"num_object_copies":-8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":-296,"num_objects_unfound":0,"num_objects_dirty":-4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002964"},"pg_stats":[{"pgid":"2.f","version":"37'10378","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.168321+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:25.168321+0000","last_peered":"2024-04-22T08:31:25.168321+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:25.168321+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40099840,"num_objects":9790,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6030},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10454","reported_seq":10569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.305659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:25.305659+0000","last_peered":"2024-04-22T08:31:25.305659+0000","last_clean":"2024-04-22T08:31:25.305659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:25.305659+0000","last_undegraded":"2024-04-22T08:31:25.305659+0000","last_fullsized":"2024-04-22T08:31:25.305659+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40050688,"num_objects":9778,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10229","reported_seq":10381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.363217+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:25.363217+0000","last_peered":"2024-04-22T08:31:25.363217+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:25.363217+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460888,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7056,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3460},{"shards":"2","objects":5894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10356","reported_seq":10386,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.354457+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:25.354457+0000","last_peered":"2024-04-22T08:31:25.354457+0000","last_clean":"2024-04-22T08:31:25.354457+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:25.354457+0000","last_undegraded":"2024-04-22T08:31:25.354457+0000","last_fullsized":"2024-04-22T08:31:25.354457+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":9758,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10267","reported_seq":10406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.199860+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:25.199860+0000","last_peered":"2024-04-22T08:31:25.199860+0000","last_clean":"2024-04-22T08:31:25.199860+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:25.199860+0000","last_undegraded":"2024-04-22T08:31:25.199860+0000","last_fullsized":"2024-04-22T08:31:25.199860+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39534592,"num_objects":9652,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10318","reported_seq":10550,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.158523+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:25.158523+0000","last_peered":"2024-04-22T08:31:25.158523+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:25.158523+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39849984,"num_objects":9729,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7187,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5885},{"shards":"0,2","objects":3587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10330","reported_seq":18066,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:28.861509+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:28.861509+0000","last_peered":"2024-04-22T08:31:28.861509+0000","last_clean":"2024-04-22T08:31:28.861509+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:28.861509+0000","last_undegraded":"2024-04-22T08:31:28.861509+0000","last_fullsized":"2024-04-22T08:31:28.861509+0000","mapping_epoch":36,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39051264,"num_objects":9534,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10219","reported_seq":10364,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:25.222255+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:25.222255+0000","last_peered":"2024-04-22T08:31:25.222255+0000","last_clean":"2024-04-22T08:31:25.222255+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:25.222255+0000","last_undegraded":"2024-04-22T08:31:25.222255+0000","last_fullsized":"2024-04-22T08:31:25.222255+0000","mapping_epoch":27,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10461","reported_seq":10616,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.382201+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:25.382201+0000","last_peered":"2024-04-22T08:31:25.382201+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:25.382201+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39978194,"num_objects":9762,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3570},{"shards":"2","objects":5890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10382","reported_seq":10518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.129634+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:25.129634+0000","last_peered":"2024-04-22T08:31:25.129634+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:25.129634+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40158418,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6064},{"shards":"0,2","objects":3486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10279","reported_seq":10448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.288380+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:25.288380+0000","last_peered":"2024-04-22T08:31:25.288380+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:25.288380+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39526400,"num_objects":9650,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3438},{"shards":"2","objects":5911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10293","reported_seq":13760,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:25.402601+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:25.402601+0000","last_peered":"2024-04-22T08:31:25.402601+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:25.402601+0000","last_undegraded":"2024-04-22T08:31:25.402601+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'200","ondisk_log_start":"28'200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39490770,"num_objects":9643,"num_object_clones":0,"num_object_copies":19286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":9643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":6983680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2098},{"shards":"2","objects":7545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10372","reported_seq":10510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.120299+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:25.120299+0000","last_peered":"2024-04-22T08:31:25.120299+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:25.120299+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40018537,"num_objects":9771,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10196","reported_seq":10419,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.178319+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:25.178319+0000","last_peered":"2024-04-22T08:31:25.178319+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:25.178319+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'100","ondisk_log_start":"28'100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5741},{"shards":"0,2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10343","reported_seq":10464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.187983+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:25.187983+0000","last_peered":"2024-04-22T08:31:25.187983+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:25.187983+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39711954,"num_objects":9697,"num_object_clones":0,"num_object_copies":19394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6014},{"shards":"0,2","objects":3383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10327","reported_seq":10527,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:25.079072+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:25.079072+0000","last_peered":"2024-04-22T08:31:25.079072+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:25.079072+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39631268,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5826},{"shards":"0,2","objects":3561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634933101,"num_objects":155025,"num_object_clones":0,"num_object_copies":310050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37757,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":155025,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138679,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5750,"num_bytes_recovered":22487657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305251840,"data_stored":1305163679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117049,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702004,"kb_used_data":431660,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93669836,"statfs":{"total":96636764160,"available":95917912064,"internally_reserved":0,"allocated":442019840,"data_stored":441804358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276819663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,0,0,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":4.3609999999999998},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.252}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":4.4050000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920812,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597528,"kb_used_data":317328,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93774312,"statfs":{"total":96636764160,"available":96024895488,"internally_reserved":0,"allocated":324943872,"data_stored":324658810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34499999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313277,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865148,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93506692,"statfs":{"total":96636764160,"available":95750852608,"internally_reserved":0,"allocated":539414528,"data_stored":539209580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,6,6,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.2610000000000001},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.2999999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539037696,"data_stored":539034217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324575232,"data_stored":324498101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":441638912,"data_stored":441631361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:31.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:31.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10378", 'reported_seq': 10510, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.168321+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:25.168321+0000', 'last_peered': '2024-04-22T08:31:25.168321+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:25.168321+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40099840, 'num_objects': 9790, 'num_object_clones': 0, 'num_object_copies': 19580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10378, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6030}, {'shards': '0,2', 'objects': 3500}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10229", 'reported_seq': 10381, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.363217+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:25.363217+0000', 'last_peered': '2024-04-22T08:31:25.363217+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:25.363217+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39460888, 'num_objects': 9635, 'num_object_clones': 0, 'num_object_copies': 19270, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3742, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9635, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7056, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3460}, {'shards': '2', 'objects': 5894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10318", 'reported_seq': 10550, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.158523+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:25.158523+0000', 'last_peered': '2024-04-22T08:31:25.158523+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:25.158523+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39849984, 'num_objects': 9729, 'num_object_clones': 0, 'num_object_copies': 19458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3846, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7187, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5885}, {'shards': '0,2', 'objects': 3587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10461", 'reported_seq': 10616, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.382201+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:25.382201+0000', 'last_peered': '2024-04-22T08:31:25.382201+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:25.382201+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39978194, 'num_objects': 9762, 'num_object_clones': 0, 'num_object_copies': 19524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3874, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10401, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3570}, {'shards': '2', 'objects': 5890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10382", 'reported_seq': 10518, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.129634+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:25.129634+0000', 'last_peered': '2024-04-22T08:31:25.129634+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:25.129634+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40158418, 'num_objects': 9806, 'num_object_clones': 0, 'num_object_copies': 19612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3744, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10380, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6064}, {'shards': '0,2', 'objects': 3486}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:31.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10279", 'reported_seq': 10448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.288380+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:25.288380+0000', 'last_peered': '2024-04-22T08:31:25.288380+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:25.288380+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39526400, 'num_objects': 9650, 'num_object_clones': 0, 'num_object_copies': 19300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3741, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6733, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3438}, {'shards': '2', 'objects': 5911}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10293", 'reported_seq': 13760, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:25.402601+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:25.402601+0000', 'last_peered': '2024-04-22T08:31:25.402601+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:25.402601+0000', 'last_undegraded': '2024-04-22T08:31:25.402601+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39490770, 'num_objects': 9643, 'num_object_clones': 0, 'num_object_copies': 19286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2098, 'num_objects_unfound': 0, 'num_objects_dirty': 9643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10291, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1707, 'num_bytes_recovered': 6983680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2098}, {'shards': '2', 'objects': 7545}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10372", 'reported_seq': 10510, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.120299+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:25.120299+0000', 'last_peered': '2024-04-22T08:31:25.120299+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:25.120299+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40018537, 'num_objects': 9771, 'num_object_clones': 0, 'num_object_copies': 19542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3693, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10371, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6079}, {'shards': '0,2', 'objects': 3421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'10196", 'reported_seq': 10419, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.178319+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:25.178319+0000', 'last_peered': '2024-04-22T08:31:25.178319+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:25.178319+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39133184, 'num_objects': 9554, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3816, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6427, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5741}, {'shards': '0,2', 'objects': 3520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10343", 'reported_seq': 10464, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.187983+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:25.187983+0000', 'last_peered': '2024-04-22T08:31:25.187983+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:25.187983+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39711954, 'num_objects': 9697, 'num_object_clones': 0, 'num_object_copies': 19394, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3685, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10341, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6014}, {'shards': '0,2', 'objects': 3383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:31.572 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10327", 'reported_seq': 10527, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:25.079072+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:25.079072+0000', 'last_peered': '2024-04-22T08:31:25.079072+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:25.079072+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39631268, 'num_objects': 9679, 'num_object_clones': 0, 'num_object_copies': 19358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3856, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10324, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5826}, {'shards': '0,2', 'objects': 3561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:31.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:32.510 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37663/306212 objects degraded (12.300%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":327,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":153106,"data_bytes":627072877,"bytes_used":2214871040,"bytes_avail":287695421440,"bytes_total":289910292480,"degraded_objects":37663,"degraded_total":306212,"degraded_ratio":0.12299648609460112,"misplaced_objects":1936,"misplaced_total":306212,"misplaced_ratio":0.0063224171489033743,"recovering_objects_per_sec":57,"recovering_bytes_per_sec":156291,"recovering_keys_per_sec":0,"num_objects_recovered":692,"num_bytes_recovered":1875968,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":398},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:32.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:32.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:33.418 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:33.420 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:33.473 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-04-22T08:31:32.241146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":627072877,"num_objects":153106,"num_object_clones":0,"num_object_copies":306212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37663,"num_objects_misplaced":1936,"num_objects_unfound":0,"num_objects_dirty":153106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140598,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6006,"num_bytes_recovered":23151209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2162960,"kb_used_data":1265848,"kb_used_omap":12,"kb_used_meta":897075,"kb_avail":280952560,"statfs":{"total":289910292480,"available":287695421440,"internally_reserved":0,"allocated":1296228352,"data_stored":1295522860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":918604910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,6,7,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19595264,"num_objects":-4784,"num_object_clones":0,"num_object_copies":-9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":-458,"num_objects_unfound":0,"num_objects_dirty":-4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":692,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002985"},"pg_stats":[{"pgid":"2.f","version":"37'10523","reported_seq":10679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.177524+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:30.177524+0000","last_peered":"2024-04-22T08:31:30.177524+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:30.177524+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39505920,"num_objects":9645,"num_object_clones":0,"num_object_copies":19290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5897},{"shards":"0,2","objects":3355}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10573","reported_seq":10688,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:30.387309+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:30.387309+0000","last_peered":"2024-04-22T08:31:30.387309+0000","last_clean":"2024-04-22T08:31:30.387309+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:30.387309+0000","last_undegraded":"2024-04-22T08:31:30.387309+0000","last_fullsized":"2024-04-22T08:31:30.387309+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39563264,"num_objects":9659,"num_object_clones":0,"num_object_copies":19318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10334","reported_seq":10492,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.286407+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:30.286407+0000","last_peered":"2024-04-22T08:31:30.286407+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:30.286407+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39030808,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7161,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3355},{"shards":"2","objects":5792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10484","reported_seq":10514,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:30.406099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:30.406099+0000","last_peered":"2024-04-22T08:31:30.406099+0000","last_clean":"2024-04-22T08:31:30.406099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:30.406099+0000","last_undegraded":"2024-04-22T08:31:30.406099+0000","last_fullsized":"2024-04-22T08:31:30.406099+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39444480,"num_objects":9630,"num_object_clones":0,"num_object_copies":19260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10484,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10382","reported_seq":10521,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:30.167727+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:30.167727+0000","last_peered":"2024-04-22T08:31:30.167727+0000","last_clean":"2024-04-22T08:31:30.167727+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:30.167727+0000","last_undegraded":"2024-04-22T08:31:30.167727+0000","last_fullsized":"2024-04-22T08:31:30.167727+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39063552,"num_objects":9537,"num_object_clones":0,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10444","reported_seq":10693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.186817+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:30.186817+0000","last_peered":"2024-04-22T08:31:30.186817+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:30.186817+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39333888,"num_objects":9603,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5768},{"shards":"0,2","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10330","reported_seq":18066,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:28.861509+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:28.861509+0000","last_peered":"2024-04-22T08:31:28.861509+0000","last_clean":"2024-04-22T08:31:28.861509+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:28.861509+0000","last_undegraded":"2024-04-22T08:31:28.861509+0000","last_fullsized":"2024-04-22T08:31:28.861509+0000","mapping_epoch":36,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39051264,"num_objects":9534,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10340","reported_seq":10485,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:30.083512+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:30.083512+0000","last_peered":"2024-04-22T08:31:30.083512+0000","last_clean":"2024-04-22T08:31:30.083512+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:30.083512+0000","last_undegraded":"2024-04-22T08:31:30.083512+0000","last_fullsized":"2024-04-22T08:31:30.083512+0000","mapping_epoch":27,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38772736,"num_objects":9466,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6988,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10597","reported_seq":10762,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.081580+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:30.081580+0000","last_peered":"2024-04-22T08:31:30.081580+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:30.081580+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39421138,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10537,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3434},{"shards":"2","objects":5759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10525","reported_seq":10684,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.195933+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:30.195933+0000","last_peered":"2024-04-22T08:31:30.195933+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:30.195933+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39572690,"num_objects":9663,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5934},{"shards":"0,2","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10411","reported_seq":10606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.405177+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:30.405177+0000","last_peered":"2024-04-22T08:31:30.405177+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:30.405177+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38985728,"num_objects":9518,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6865,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3306},{"shards":"2","objects":5792}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10424","reported_seq":14215,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:30.408880+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:30.408880+0000","last_peered":"2024-04-22T08:31:30.408880+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:30.408880+0000","last_undegraded":"2024-04-22T08:31:30.408880+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38954194,"num_objects":9512,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1936,"num_objects_unfound":0,"num_objects_dirty":9512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1869,"num_bytes_recovered":7647232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1936},{"shards":"2","objects":7576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10488","reported_seq":10649,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.233948+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:30.233948+0000","last_peered":"2024-04-22T08:31:30.233948+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:30.233948+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39543401,"num_objects":9655,"num_object_clones":0,"num_object_copies":19310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5975},{"shards":"0,2","objects":3305}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10326","reported_seq":10558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.187332+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:30.187332+0000","last_peered":"2024-04-22T08:31:30.187332+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:30.187332+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'300","ondisk_log_start":"28'300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":9424,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6557,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5616},{"shards":"0,2","objects":3390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10473","reported_seq":10619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.190734+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:30.190734+0000","last_peered":"2024-04-22T08:31:30.190734+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:30.190734+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39179474,"num_objects":9567,"num_object_clones":0,"num_object_copies":19134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10471,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5897},{"shards":"0,2","objects":3253}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10469","reported_seq":10687,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:30.121272+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:30.121272+0000","last_peered":"2024-04-22T08:31:30.121272+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:30.121272+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39049636,"num_objects":9537,"num_object_clones":0,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5693},{"shards":"0,2","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":627072877,"num_objects":153106,"num_object_clones":0,"num_object_copies":306212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37663,"num_objects_misplaced":1936,"num_objects_unfound":0,"num_objects_dirty":153106,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140598,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6006,"num_bytes_recovered":23151209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294950400,"data_stored":1294862239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117050,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":427760,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":438026240,"data_stored":437810758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281013967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.32200000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920812,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597528,"kb_used_data":317328,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93774312,"statfs":{"total":96636764160,"available":96024895488,"internally_reserved":0,"allocated":324943872,"data_stored":324658810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34499999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313278,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":863232,"kb_used_data":520760,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93508608,"statfs":{"total":96636764160,"available":95752814592,"internally_reserved":0,"allocated":533258240,"data_stored":533053292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,3,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.39300000000000002},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":1.8169999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532791296,"data_stored":532787817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324575232,"data_stored":324498101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":437583872,"data_stored":437576321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:36.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:37.288 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:37.366 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-22T08:31:36.243983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2152372,"kb_used_data":1255260,"kb_used_omap":12,"kb_used_meta":897075,"kb_avail":280963148,"statfs":{"total":289910292480,"available":287706263552,"internally_reserved":0,"allocated":1285386240,"data_stored":1284680748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":918604910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,2,5,7,2,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24477696,"num_objects":-5976,"num_object_clones":0,"num_object_copies":-11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":-491,"num_objects_unfound":0,"num_objects_dirty":-5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":2011136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004843"},"pg_stats":[{"pgid":"2.f","version":"37'10647","reported_seq":10815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.268345+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:35.268345+0000","last_peered":"2024-04-22T08:31:35.268345+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:35.268345+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38998016,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5779},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10702","reported_seq":10817,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437431+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:35.437431+0000","last_peered":"2024-04-22T08:31:35.437431+0000","last_clean":"2024-04-22T08:31:35.437431+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:35.437431+0000","last_undegraded":"2024-04-22T08:31:35.437431+0000","last_fullsized":"2024-04-22T08:31:35.437431+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10473","reported_seq":10639,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.448253+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:35.448253+0000","last_peered":"2024-04-22T08:31:35.448253+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:35.448253+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461464,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7300,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3216},{"shards":"2","objects":5657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10628","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437416+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:35.437416+0000","last_peered":"2024-04-22T08:31:35.437416+0000","last_clean":"2024-04-22T08:31:35.437416+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:35.437416+0000","last_undegraded":"2024-04-22T08:31:35.437416+0000","last_fullsized":"2024-04-22T08:31:35.437416+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38854656,"num_objects":9486,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10520","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.293325+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:35.293325+0000","last_peered":"2024-04-22T08:31:35.293325+0000","last_clean":"2024-04-22T08:31:35.293325+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:35.293325+0000","last_undegraded":"2024-04-22T08:31:35.293325+0000","last_fullsized":"2024-04-22T08:31:35.293325+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38498304,"num_objects":9399,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10576","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.231832+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:35.231832+0000","last_peered":"2024-04-22T08:31:35.231832+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:35.231832+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38793216,"num_objects":9471,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5649},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10459","reported_seq":18195,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:33.925258+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:33.925258+0000","last_peered":"2024-04-22T08:31:33.925258+0000","last_clean":"2024-04-22T08:31:33.925258+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:33.925258+0000","last_undegraded":"2024-04-22T08:31:33.925258+0000","last_fullsized":"2024-04-22T08:31:33.925258+0000","mapping_epoch":36,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38522880,"num_objects":9405,"num_object_clones":0,"num_object_copies":18810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10454","reported_seq":10599,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.243572+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:35.243572+0000","last_peered":"2024-04-22T08:31:35.243572+0000","last_clean":"2024-04-22T08:31:35.243572+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:35.243572+0000","last_undegraded":"2024-04-22T08:31:35.243572+0000","last_fullsized":"2024-04-22T08:31:35.243572+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7102,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10729","reported_seq":10904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.403652+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:35.403652+0000","last_peered":"2024-04-22T08:31:35.403652+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:35.403652+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38880466,"num_objects":9494,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3302},{"shards":"2","objects":5632}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10656","reported_seq":10836,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.143815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:35.143815+0000","last_peered":"2024-04-22T08:31:35.143815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:35.143815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39036114,"num_objects":9532,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10654,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":3212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10513","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.410118+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:35.410118+0000","last_peered":"2024-04-22T08:31:35.410118+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:35.410118+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3204},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10530","reported_seq":14657,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:35.432551+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:35.432551+0000","last_peered":"2024-04-22T08:31:35.432551+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:35.432551+0000","last_undegraded":"2024-04-22T08:31:35.432551+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38520018,"num_objects":9406,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":9406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8335360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1768},{"shards":"2","objects":7638}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10619","reported_seq":10797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.213523+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:35.213523+0000","last_peered":"2024-04-22T08:31:35.213523+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:35.213523+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39006825,"num_objects":9524,"num_object_clones":0,"num_object_copies":19048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10618,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5852},{"shards":"0,2","objects":3174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10473","reported_seq":10718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.276338+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:35.276338+0000","last_peered":"2024-04-22T08:31:35.276338+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:35.276338+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37998592,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6704,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5475},{"shards":"0,2","objects":3243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10603","reported_seq":10758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.271795+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:35.271795+0000","last_peered":"2024-04-22T08:31:35.271795+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:35.271795+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38646994,"num_objects":9437,"num_object_clones":0,"num_object_copies":18874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5771},{"shards":"0,2","objects":3123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10602","reported_seq":10842,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.063527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:35.063527+0000","last_peered":"2024-04-22T08:31:35.063527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:35.063527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38504868,"num_objects":9404,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5571},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284079616,"data_stored":1283991455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685964,"kb_used_data":423812,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93685876,"statfs":{"total":96636764160,"available":95934337024,"internally_reserved":0,"allocated":433983488,"data_stored":433768006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,5,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":7.3440000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920813,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605464,"kb_used_data":317072,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93766376,"statfs":{"total":96636764160,"available":96016769024,"internally_reserved":0,"allocated":324681728,"data_stored":324396666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.252},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.222}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":7.1340000000000003},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":7.1669999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313279,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":860944,"kb_used_data":514376,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93510896,"statfs":{"total":96636764160,"available":95755157504,"internally_reserved":0,"allocated":526721024,"data_stored":526516076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,0,3,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.314},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526282752,"data_stored":526279273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324313088,"data_stored":324235957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":433483776,"data_stored":433476225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:37.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:38.065 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:38.067 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:38.133 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-22T08:31:36.243983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2152372,"kb_used_data":1255260,"kb_used_omap":12,"kb_used_meta":897075,"kb_avail":280963148,"statfs":{"total":289910292480,"available":287706263552,"internally_reserved":0,"allocated":1285386240,"data_stored":1284680748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":918604910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,2,5,7,2,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24477696,"num_objects":-5976,"num_object_clones":0,"num_object_copies":-11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":-491,"num_objects_unfound":0,"num_objects_dirty":-5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":2011136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004843"},"pg_stats":[{"pgid":"2.f","version":"37'10647","reported_seq":10815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.268345+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:35.268345+0000","last_peered":"2024-04-22T08:31:35.268345+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:35.268345+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38998016,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5779},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10702","reported_seq":10817,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437431+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:35.437431+0000","last_peered":"2024-04-22T08:31:35.437431+0000","last_clean":"2024-04-22T08:31:35.437431+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:35.437431+0000","last_undegraded":"2024-04-22T08:31:35.437431+0000","last_fullsized":"2024-04-22T08:31:35.437431+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10473","reported_seq":10639,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.448253+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:35.448253+0000","last_peered":"2024-04-22T08:31:35.448253+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:35.448253+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461464,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7300,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3216},{"shards":"2","objects":5657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10628","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437416+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:35.437416+0000","last_peered":"2024-04-22T08:31:35.437416+0000","last_clean":"2024-04-22T08:31:35.437416+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:35.437416+0000","last_undegraded":"2024-04-22T08:31:35.437416+0000","last_fullsized":"2024-04-22T08:31:35.437416+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38854656,"num_objects":9486,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10520","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.293325+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:35.293325+0000","last_peered":"2024-04-22T08:31:35.293325+0000","last_clean":"2024-04-22T08:31:35.293325+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:35.293325+0000","last_undegraded":"2024-04-22T08:31:35.293325+0000","last_fullsized":"2024-04-22T08:31:35.293325+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38498304,"num_objects":9399,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10576","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.231832+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:35.231832+0000","last_peered":"2024-04-22T08:31:35.231832+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:35.231832+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38793216,"num_objects":9471,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5649},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10459","reported_seq":18195,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:33.925258+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:33.925258+0000","last_peered":"2024-04-22T08:31:33.925258+0000","last_clean":"2024-04-22T08:31:33.925258+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:33.925258+0000","last_undegraded":"2024-04-22T08:31:33.925258+0000","last_fullsized":"2024-04-22T08:31:33.925258+0000","mapping_epoch":36,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38522880,"num_objects":9405,"num_object_clones":0,"num_object_copies":18810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10454","reported_seq":10599,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.243572+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:35.243572+0000","last_peered":"2024-04-22T08:31:35.243572+0000","last_clean":"2024-04-22T08:31:35.243572+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:35.243572+0000","last_undegraded":"2024-04-22T08:31:35.243572+0000","last_fullsized":"2024-04-22T08:31:35.243572+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7102,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10729","reported_seq":10904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.403652+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:35.403652+0000","last_peered":"2024-04-22T08:31:35.403652+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:35.403652+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38880466,"num_objects":9494,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3302},{"shards":"2","objects":5632}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10656","reported_seq":10836,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.143815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:35.143815+0000","last_peered":"2024-04-22T08:31:35.143815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:35.143815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39036114,"num_objects":9532,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10654,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":3212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10513","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.410118+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:35.410118+0000","last_peered":"2024-04-22T08:31:35.410118+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:35.410118+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3204},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10530","reported_seq":14657,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:35.432551+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:35.432551+0000","last_peered":"2024-04-22T08:31:35.432551+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:35.432551+0000","last_undegraded":"2024-04-22T08:31:35.432551+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38520018,"num_objects":9406,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":9406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8335360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1768},{"shards":"2","objects":7638}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10619","reported_seq":10797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.213523+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:35.213523+0000","last_peered":"2024-04-22T08:31:35.213523+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:35.213523+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39006825,"num_objects":9524,"num_object_clones":0,"num_object_copies":19048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10618,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5852},{"shards":"0,2","objects":3174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10473","reported_seq":10718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.276338+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:35.276338+0000","last_peered":"2024-04-22T08:31:35.276338+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:35.276338+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37998592,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6704,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5475},{"shards":"0,2","objects":3243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10603","reported_seq":10758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.271795+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:35.271795+0000","last_peered":"2024-04-22T08:31:35.271795+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:35.271795+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38646994,"num_objects":9437,"num_object_clones":0,"num_object_copies":18874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5771},{"shards":"0,2","objects":3123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10602","reported_seq":10842,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.063527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:35.063527+0000","last_peered":"2024-04-22T08:31:35.063527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:35.063527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38504868,"num_objects":9404,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5571},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284079616,"data_stored":1283991455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685964,"kb_used_data":423812,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93685876,"statfs":{"total":96636764160,"available":95934337024,"internally_reserved":0,"allocated":433983488,"data_stored":433768006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,5,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":7.3440000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920813,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605464,"kb_used_data":317072,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93766376,"statfs":{"total":96636764160,"available":96016769024,"internally_reserved":0,"allocated":324681728,"data_stored":324396666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.252},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.222}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":7.1340000000000003},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":7.1669999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313279,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":860944,"kb_used_data":514376,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93510896,"statfs":{"total":96636764160,"available":95755157504,"internally_reserved":0,"allocated":526721024,"data_stored":526516076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,0,3,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.314},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526282752,"data_stored":526279273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324313088,"data_stored":324235957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":433483776,"data_stored":433476225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10647", 'reported_seq': 10815, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.268345+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:35.268345+0000', 'last_peered': '2024-04-22T08:31:35.268345+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:35.268345+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38998016, 'num_objects': 9521, 'num_object_clones': 0, 'num_object_copies': 19042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3742, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10647, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5779}, {'shards': '0,2', 'objects': 3231}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10473", 'reported_seq': 10639, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.448253+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:35.448253+0000', 'last_peered': '2024-04-22T08:31:35.448253+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:35.448253+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38461464, 'num_objects': 9391, 'num_object_clones': 0, 'num_object_copies': 18782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3735, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9391, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7300, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3216}, {'shards': '2', 'objects': 5657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10576", 'reported_seq': 10850, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.231832+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:35.231832+0000', 'last_peered': '2024-04-22T08:31:35.231832+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:35.231832+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38793216, 'num_objects': 9471, 'num_object_clones': 0, 'num_object_copies': 18942, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3824, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7445, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5649}, {'shards': '0,2', 'objects': 3329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:38.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10729", 'reported_seq': 10904, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.403652+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:35.403652+0000', 'last_peered': '2024-04-22T08:31:35.403652+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:35.403652+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38880466, 'num_objects': 9494, 'num_object_clones': 0, 'num_object_copies': 18988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10669, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3302}, {'shards': '2', 'objects': 5632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10656", 'reported_seq': 10836, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.143815+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:35.143815+0000', 'last_peered': '2024-04-22T08:31:35.143815+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:35.143815+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39036114, 'num_objects': 9532, 'num_object_clones': 0, 'num_object_copies': 19064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3722, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10654, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5812}, {'shards': '0,2', 'objects': 3212}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10513", 'reported_seq': 10728, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.410118+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:35.410118+0000', 'last_peered': '2024-04-22T08:31:35.410118+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:35.410118+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38567936, 'num_objects': 9416, 'num_object_clones': 0, 'num_object_copies': 18832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3718, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6967, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3204}, {'shards': '2', 'objects': 5700}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10530", 'reported_seq': 14657, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:35.432551+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:35.432551+0000', 'last_peered': '2024-04-22T08:31:35.432551+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:35.432551+0000', 'last_undegraded': '2024-04-22T08:31:35.432551+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38520018, 'num_objects': 9406, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1768, 'num_objects_unfound': 0, 'num_objects_dirty': 9406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10528, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2037, 'num_bytes_recovered': 8335360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1768}, {'shards': '2', 'objects': 7638}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10619", 'reported_seq': 10797, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.213523+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:35.213523+0000', 'last_peered': '2024-04-22T08:31:35.213523+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:35.213523+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39006825, 'num_objects': 9524, 'num_object_clones': 0, 'num_object_copies': 19048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3673, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10618, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5852}, {'shards': '0,2', 'objects': 3174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:38.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'10473", 'reported_seq': 10718, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.276338+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:35.276338+0000', 'last_peered': '2024-04-22T08:31:35.276338+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:35.276338+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37998592, 'num_objects': 9277, 'num_object_clones': 0, 'num_object_copies': 18554, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3805, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6704, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5475}, {'shards': '0,2', 'objects': 3243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.137 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:38.137 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10603", 'reported_seq': 10758, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.271795+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:35.271795+0000', 'last_peered': '2024-04-22T08:31:35.271795+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:35.271795+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38646994, 'num_objects': 9437, 'num_object_clones': 0, 'num_object_copies': 18874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3668, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10601, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5771}, {'shards': '0,2', 'objects': 3123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.137 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:38.137 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10602", 'reported_seq': 10842, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:35.063527+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:35.063527+0000', 'last_peered': '2024-04-22T08:31:35.063527+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:35.063527+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38504868, 'num_objects': 9404, 'num_object_clones': 0, 'num_object_copies': 18808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3836, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10599, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5571}, {'shards': '0,2', 'objects': 3286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:38.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:39.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:39.051 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37587/302090 objects degraded (12.442%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":333,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":151045,"data_bytes":618631021,"bytes_used":2204028928,"bytes_avail":287706263552,"bytes_total":289910292480,"degraded_objects":37587,"degraded_total":302090,"degraded_ratio":0.1244231851435003,"misplaced_objects":1768,"misplaced_total":302090,"misplaced_ratio":0.0058525604952166573,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":112594,"recovering_keys_per_sec":0,"num_objects_recovered":500,"num_bytes_recovered":1351680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":340},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:39.085 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:39.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:39.669 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:39.671 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:39.737 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-22T08:31:38.244496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2152372,"kb_used_data":1255260,"kb_used_omap":12,"kb_used_meta":897075,"kb_avail":280963148,"statfs":{"total":289910292480,"available":287706263552,"internally_reserved":0,"allocated":1285386240,"data_stored":1284680748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":918604910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,2,5,7,2,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16732160,"num_objects":-4085,"num_object_clones":0,"num_object_copies":-8170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":-330,"num_objects_unfound":0,"num_objects_dirty":-4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":1351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004868"},"pg_stats":[{"pgid":"2.f","version":"37'10647","reported_seq":10815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.268345+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:35.268345+0000","last_peered":"2024-04-22T08:31:35.268345+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:35.268345+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38998016,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5779},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10702","reported_seq":10817,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437431+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:35.437431+0000","last_peered":"2024-04-22T08:31:35.437431+0000","last_clean":"2024-04-22T08:31:35.437431+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:35.437431+0000","last_undegraded":"2024-04-22T08:31:35.437431+0000","last_fullsized":"2024-04-22T08:31:35.437431+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10473","reported_seq":10639,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.448253+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:35.448253+0000","last_peered":"2024-04-22T08:31:35.448253+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:35.448253+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461464,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7300,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3216},{"shards":"2","objects":5657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10628","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.437416+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:35.437416+0000","last_peered":"2024-04-22T08:31:35.437416+0000","last_clean":"2024-04-22T08:31:35.437416+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:35.437416+0000","last_undegraded":"2024-04-22T08:31:35.437416+0000","last_fullsized":"2024-04-22T08:31:35.437416+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38854656,"num_objects":9486,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10520","reported_seq":10659,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.293325+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:35.293325+0000","last_peered":"2024-04-22T08:31:35.293325+0000","last_clean":"2024-04-22T08:31:35.293325+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:35.293325+0000","last_undegraded":"2024-04-22T08:31:35.293325+0000","last_fullsized":"2024-04-22T08:31:35.293325+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38498304,"num_objects":9399,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10576","reported_seq":10850,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.231832+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:35.231832+0000","last_peered":"2024-04-22T08:31:35.231832+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:35.231832+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38793216,"num_objects":9471,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5649},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10459","reported_seq":18195,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:33.925258+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:33.925258+0000","last_peered":"2024-04-22T08:31:33.925258+0000","last_clean":"2024-04-22T08:31:33.925258+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:33.925258+0000","last_undegraded":"2024-04-22T08:31:33.925258+0000","last_fullsized":"2024-04-22T08:31:33.925258+0000","mapping_epoch":36,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38522880,"num_objects":9405,"num_object_clones":0,"num_object_copies":18810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10454","reported_seq":10599,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:35.243572+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:35.243572+0000","last_peered":"2024-04-22T08:31:35.243572+0000","last_clean":"2024-04-22T08:31:35.243572+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:35.243572+0000","last_undegraded":"2024-04-22T08:31:35.243572+0000","last_fullsized":"2024-04-22T08:31:35.243572+0000","mapping_epoch":27,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7102,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10729","reported_seq":10904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.403652+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:35.403652+0000","last_peered":"2024-04-22T08:31:35.403652+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:35.403652+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38880466,"num_objects":9494,"num_object_clones":0,"num_object_copies":18988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3302},{"shards":"2","objects":5632}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10656","reported_seq":10836,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.143815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:35.143815+0000","last_peered":"2024-04-22T08:31:35.143815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:35.143815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39036114,"num_objects":9532,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10654,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":3212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10513","reported_seq":10728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.410118+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:35.410118+0000","last_peered":"2024-04-22T08:31:35.410118+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:35.410118+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3204},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10530","reported_seq":14657,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:35.432551+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:35.432551+0000","last_peered":"2024-04-22T08:31:35.432551+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:35.432551+0000","last_undegraded":"2024-04-22T08:31:35.432551+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38520018,"num_objects":9406,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":9406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8335360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1768},{"shards":"2","objects":7638}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10619","reported_seq":10797,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.213523+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:35.213523+0000","last_peered":"2024-04-22T08:31:35.213523+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:35.213523+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39006825,"num_objects":9524,"num_object_clones":0,"num_object_copies":19048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10618,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5852},{"shards":"0,2","objects":3174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10473","reported_seq":10718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.276338+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:35.276338+0000","last_peered":"2024-04-22T08:31:35.276338+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:35.276338+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'400","ondisk_log_start":"28'400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37998592,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6704,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5475},{"shards":"0,2","objects":3243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10603","reported_seq":10758,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.271795+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:35.271795+0000","last_peered":"2024-04-22T08:31:35.271795+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:35.271795+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38646994,"num_objects":9437,"num_object_clones":0,"num_object_copies":18874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5771},{"shards":"0,2","objects":3123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10602","reported_seq":10842,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:35.063527+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:35.063527+0000","last_peered":"2024-04-22T08:31:35.063527+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:35.063527+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38504868,"num_objects":9404,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5571},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":618631021,"num_objects":151045,"num_object_clones":0,"num_object_copies":302090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37587,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":151045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142659,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6250,"num_bytes_recovered":23839337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284079616,"data_stored":1283991455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685964,"kb_used_data":423812,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93685876,"statfs":{"total":96636764160,"available":95934337024,"internally_reserved":0,"allocated":433983488,"data_stored":433768006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,5,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":7.3440000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920813,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605464,"kb_used_data":317072,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93766376,"statfs":{"total":96636764160,"available":96016769024,"internally_reserved":0,"allocated":324681728,"data_stored":324396666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.252},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.222}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":7.1340000000000003},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":7.1669999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313279,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":860944,"kb_used_data":514376,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93510896,"statfs":{"total":96636764160,"available":95755157504,"internally_reserved":0,"allocated":526721024,"data_stored":526516076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,0,3,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.314},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526282752,"data_stored":526279273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324313088,"data_stored":324235957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":433483776,"data_stored":433476225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:42.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:43.565 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:43.567 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:43.635 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-22T08:31:42.245491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":610430829,"num_objects":149043,"num_object_clones":0,"num_object_copies":298086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":149043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144661,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160686,"ondisk_log_size":160686,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2159828,"kb_used_data":1244664,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":280955692,"statfs":{"total":289910292480,"available":287698628608,"internally_reserved":0,"allocated":1274535936,"data_stored":1273837402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":937086062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,10,5,9,5,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24502272,"num_objects":-5982,"num_object_clones":0,"num_object_copies":-11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":-493,"num_objects_unfound":0,"num_objects_dirty":-5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":2019328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004811"},"pg_stats":[{"pgid":"2.f","version":"37'10774","reported_seq":10984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.325556+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:40.325556+0000","last_peered":"2024-04-22T08:31:40.325556+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:40.325556+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":9394,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5673},{"shards":"0,2","objects":3104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10829","reported_seq":10944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.390153+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:40.390153+0000","last_peered":"2024-04-22T08:31:40.390153+0000","last_clean":"2024-04-22T08:31:40.390153+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:40.390153+0000","last_undegraded":"2024-04-22T08:31:40.390153+0000","last_fullsized":"2024-04-22T08:31:40.390153+0000","mapping_epoch":27,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514688,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7885,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10600","reported_seq":10769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.372729+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:40.372729+0000","last_peered":"2024-04-22T08:31:40.372729+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:40.372729+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37941272,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7427,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3089},{"shards":"2","objects":5531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10761","reported_seq":10792,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.405369+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:40.405369+0000","last_peered":"2024-04-22T08:31:40.405369+0000","last_clean":"2024-04-22T08:31:40.405369+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:40.405369+0000","last_undegraded":"2024-04-22T08:31:40.405369+0000","last_fullsized":"2024-04-22T08:31:40.405369+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38309888,"num_objects":9353,"num_object_clones":0,"num_object_copies":18706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10663","reported_seq":10802,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.257620+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:40.257620+0000","last_peered":"2024-04-22T08:31:40.257620+0000","last_clean":"2024-04-22T08:31:40.257620+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:40.257620+0000","last_undegraded":"2024-04-22T08:31:40.257620+0000","last_fullsized":"2024-04-22T08:31:40.257620+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37912576,"num_objects":9256,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10707","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.280590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:40.280590+0000","last_peered":"2024-04-22T08:31:40.280590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:40.280590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":9340,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10601","reported_seq":18337,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:38.920440+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:38.920440+0000","last_peered":"2024-04-22T08:31:38.920440+0000","last_clean":"2024-04-22T08:31:38.920440+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:38.920440+0000","last_undegraded":"2024-04-22T08:31:38.920440+0000","last_fullsized":"2024-04-22T08:31:38.920440+0000","mapping_epoch":36,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37941248,"num_objects":9263,"num_object_clones":0,"num_object_copies":18526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10574","reported_seq":10718,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.335504+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:40.335504+0000","last_peered":"2024-04-22T08:31:40.335504+0000","last_clean":"2024-04-22T08:31:40.335504+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:40.335504+0000","last_undegraded":"2024-04-22T08:31:40.335504+0000","last_fullsized":"2024-04-22T08:31:40.335504+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":9232,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10861","reported_seq":11036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.447032+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:40.447032+0000","last_peered":"2024-04-22T08:31:40.447032+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:40.447032+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38339794,"num_objects":9362,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3170},{"shards":"2","objects":5501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10769","reported_seq":10985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.226021+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:40.226021+0000","last_peered":"2024-04-22T08:31:40.226021+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:40.226021+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573266,"num_objects":9419,"num_object_clones":0,"num_object_copies":18838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5717},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10627","reported_seq":10846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.376745+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:40.376745+0000","last_peered":"2024-04-22T08:31:40.376745+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:40.376745+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":9302,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3090},{"shards":"2","objects":5588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10647","reported_seq":15104,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:40.452998+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:40.452998+0000","last_peered":"2024-04-22T08:31:40.452998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:40.452998+0000","last_undegraded":"2024-04-22T08:31:40.452998+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2202,"num_bytes_recovered":9003008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1605},{"shards":"2","objects":7684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10735","reported_seq":10943,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.302768+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:40.302768+0000","last_peered":"2024-04-22T08:31:40.302768+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:40.302768+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38531689,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5751},{"shards":"0,2","objects":3058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10607","reported_seq":10876,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.300082+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:40.300082+0000","last_peered":"2024-04-22T08:31:40.300082+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:40.300082+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37449728,"num_objects":9143,"num_object_clones":0,"num_object_copies":18286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":3109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10712","reported_seq":10883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.293956+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:40.293956+0000","last_peered":"2024-04-22T08:31:40.293956+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:40.293956+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200530,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10719","reported_seq":10988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.326445+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:40.326445+0000","last_peered":"2024-04-22T08:31:40.326445+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:40.326445+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38025636,"num_objects":9287,"num_object_clones":0,"num_object_copies":18574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5469},{"shards":"0,2","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":610430829,"num_objects":149043,"num_object_clones":0,"num_object_copies":298086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":149043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144661,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273307136,"data_stored":1273225933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160686,"ondisk_log_size":160686,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117052,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704044,"kb_used_data":419748,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93667796,"statfs":{"total":96636764160,"available":95915823104,"internally_reserved":0,"allocated":429821952,"data_stored":429606470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":3.0070000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":3.044}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920814,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609108,"kb_used_data":316620,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93762732,"statfs":{"total":96636764160,"available":96013037568,"internally_reserved":0,"allocated":324218880,"data_stored":323940776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":299495116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":2.097},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":2.1459999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":2.1850000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":2.2080000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313280,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":846676,"kb_used_data":508296,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93525164,"statfs":{"total":96636764160,"available":95769767936,"internally_reserved":0,"allocated":520495104,"data_stored":520290156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,1,1,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.5819999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.5369999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520085504,"data_stored":520082025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323895296,"data_stored":323825123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":429326336,"data_stored":429318785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:43.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:44.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:44.392 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:44.460 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-22T08:31:44.246037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609873773,"num_objects":148907,"num_object_clones":0,"num_object_copies":297814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":148907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144797,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2151388,"kb_used_data":1244412,"kb_used_omap":12,"kb_used_meta":906931,"kb_avail":280964132,"statfs":{"total":289910292480,"available":287707271168,"internally_reserved":0,"allocated":1274277888,"data_stored":1273579354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":928697454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,10,2,5,2,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17199104,"num_objects":-4199,"num_object_clones":0,"num_object_copies":-8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":-331,"num_objects_unfound":0,"num_objects_dirty":-4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":1355776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004891"},"pg_stats":[{"pgid":"2.f","version":"37'10774","reported_seq":10984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.325556+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:40.325556+0000","last_peered":"2024-04-22T08:31:40.325556+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:40.325556+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":9394,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5673},{"shards":"0,2","objects":3104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10829","reported_seq":10944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.390153+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:40.390153+0000","last_peered":"2024-04-22T08:31:40.390153+0000","last_clean":"2024-04-22T08:31:40.390153+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:40.390153+0000","last_undegraded":"2024-04-22T08:31:40.390153+0000","last_fullsized":"2024-04-22T08:31:40.390153+0000","mapping_epoch":27,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514688,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7885,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10600","reported_seq":10769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.372729+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:40.372729+0000","last_peered":"2024-04-22T08:31:40.372729+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:40.372729+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37941272,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7427,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3089},{"shards":"2","objects":5531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10761","reported_seq":10792,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.405369+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:40.405369+0000","last_peered":"2024-04-22T08:31:40.405369+0000","last_clean":"2024-04-22T08:31:40.405369+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:40.405369+0000","last_undegraded":"2024-04-22T08:31:40.405369+0000","last_fullsized":"2024-04-22T08:31:40.405369+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38309888,"num_objects":9353,"num_object_clones":0,"num_object_copies":18706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10663","reported_seq":10802,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.257620+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:40.257620+0000","last_peered":"2024-04-22T08:31:40.257620+0000","last_clean":"2024-04-22T08:31:40.257620+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:40.257620+0000","last_undegraded":"2024-04-22T08:31:40.257620+0000","last_fullsized":"2024-04-22T08:31:40.257620+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37912576,"num_objects":9256,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10707","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.280590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:40.280590+0000","last_peered":"2024-04-22T08:31:40.280590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:40.280590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":9340,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10737","reported_seq":18473,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:43.938811+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:43.938811+0000","last_peered":"2024-04-22T08:31:43.938811+0000","last_clean":"2024-04-22T08:31:43.938811+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:43.938811+0000","last_undegraded":"2024-04-22T08:31:43.938811+0000","last_fullsized":"2024-04-22T08:31:43.938811+0000","mapping_epoch":36,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37384192,"num_objects":9127,"num_object_clones":0,"num_object_copies":18254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10574","reported_seq":10718,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.335504+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:40.335504+0000","last_peered":"2024-04-22T08:31:40.335504+0000","last_clean":"2024-04-22T08:31:40.335504+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:40.335504+0000","last_undegraded":"2024-04-22T08:31:40.335504+0000","last_fullsized":"2024-04-22T08:31:40.335504+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":9232,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10861","reported_seq":11036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.447032+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:40.447032+0000","last_peered":"2024-04-22T08:31:40.447032+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:40.447032+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38339794,"num_objects":9362,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3170},{"shards":"2","objects":5501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10769","reported_seq":10985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.226021+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:40.226021+0000","last_peered":"2024-04-22T08:31:40.226021+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:40.226021+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573266,"num_objects":9419,"num_object_clones":0,"num_object_copies":18838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5717},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10627","reported_seq":10846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.376745+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:40.376745+0000","last_peered":"2024-04-22T08:31:40.376745+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:40.376745+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":9302,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3090},{"shards":"2","objects":5588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10647","reported_seq":15104,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:40.452998+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:40.452998+0000","last_peered":"2024-04-22T08:31:40.452998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:40.452998+0000","last_undegraded":"2024-04-22T08:31:40.452998+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2202,"num_bytes_recovered":9003008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1605},{"shards":"2","objects":7684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10735","reported_seq":10943,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.302768+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:40.302768+0000","last_peered":"2024-04-22T08:31:40.302768+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:40.302768+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38531689,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5751},{"shards":"0,2","objects":3058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10607","reported_seq":10876,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.300082+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:40.300082+0000","last_peered":"2024-04-22T08:31:40.300082+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:40.300082+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37449728,"num_objects":9143,"num_object_clones":0,"num_object_copies":18286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":3109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10712","reported_seq":10883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.293956+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:40.293956+0000","last_peered":"2024-04-22T08:31:40.293956+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:40.293956+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200530,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10719","reported_seq":10988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.326445+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:40.326445+0000","last_peered":"2024-04-22T08:31:40.326445+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:40.326445+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38025636,"num_objects":9287,"num_object_clones":0,"num_object_copies":18574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5469},{"shards":"0,2","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":609873773,"num_objects":148907,"num_object_clones":0,"num_object_copies":297814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":148907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144797,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273049088,"data_stored":1272967885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117052,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704044,"kb_used_data":419748,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93667796,"statfs":{"total":96636764160,"available":95915823104,"internally_reserved":0,"allocated":429821952,"data_stored":429606470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":3.0070000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":3.044}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920815,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600668,"kb_used_data":316368,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93771172,"statfs":{"total":96636764160,"available":96021680128,"internally_reserved":0,"allocated":323960832,"data_stored":323682728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":291106508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.193},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.221}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313280,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":846676,"kb_used_data":508296,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93525164,"statfs":{"total":96636764160,"available":95769767936,"internally_reserved":0,"allocated":520495104,"data_stored":520290156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,1,1,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.5819999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.5369999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520085504,"data_stored":520082025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323637248,"data_stored":323567075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":429326336,"data_stored":429318785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:44.461 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:44.461 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10774", 'reported_seq': 10984, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.325556+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:40.325556+0000', 'last_peered': '2024-04-22T08:31:40.325556+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:40.325556+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38477824, 'num_objects': 9394, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3721, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5673}, {'shards': '0,2', 'objects': 3104}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10600", 'reported_seq': 10769, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.372729+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:40.372729+0000', 'last_peered': '2024-04-22T08:31:40.372729+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:40.372729+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37941272, 'num_objects': 9264, 'num_object_clones': 0, 'num_object_copies': 18528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3734, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9264, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7427, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3089}, {'shards': '2', 'objects': 5531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10707", 'reported_seq': 11020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.280590+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:40.280590+0000', 'last_peered': '2024-04-22T08:31:40.280590+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:40.280590+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38256640, 'num_objects': 9340, 'num_object_clones': 0, 'num_object_copies': 18680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3806, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7576, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5536}, {'shards': '0,2', 'objects': 3198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10861", 'reported_seq': 11036, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.447032+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:40.447032+0000', 'last_peered': '2024-04-22T08:31:40.447032+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:40.447032+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38339794, 'num_objects': 9362, 'num_object_clones': 0, 'num_object_copies': 18724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10801, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3170}, {'shards': '2', 'objects': 5501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:44.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10769", 'reported_seq': 10985, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.226021+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:40.226021+0000', 'last_peered': '2024-04-22T08:31:40.226021+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:40.226021+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38573266, 'num_objects': 9419, 'num_object_clones': 0, 'num_object_copies': 18838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3704, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10767, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5717}, {'shards': '0,2', 'objects': 3099}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10627", 'reported_seq': 10846, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.376745+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:40.376745+0000', 'last_peered': '2024-04-22T08:31:40.376745+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:40.376745+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38100992, 'num_objects': 9302, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7081, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3090}, {'shards': '2', 'objects': 5588}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10647", 'reported_seq': 15104, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:40.452998+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:40.452998+0000', 'last_peered': '2024-04-22T08:31:40.452998+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:40.452998+0000', 'last_undegraded': '2024-04-22T08:31:40.452998+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38040786, 'num_objects': 9289, 'num_object_clones': 0, 'num_object_copies': 18578, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1605, 'num_objects_unfound': 0, 'num_objects_dirty': 9289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10645, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2202, 'num_bytes_recovered': 9003008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1605}, {'shards': '2', 'objects': 7684}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10735", 'reported_seq': 10943, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.302768+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:40.302768+0000', 'last_peered': '2024-04-22T08:31:40.302768+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:40.302768+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38531689, 'num_objects': 9408, 'num_object_clones': 0, 'num_object_copies': 18816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3658, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10734, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 66, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5751}, {'shards': '0,2', 'objects': 3058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:44.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'10607", 'reported_seq': 10876, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.300082+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:40.300082+0000', 'last_peered': '2024-04-22T08:31:40.300082+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:40.300082+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37449728, 'num_objects': 9143, 'num_object_clones': 0, 'num_object_copies': 18286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3793, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6838, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5353}, {'shards': '0,2', 'objects': 3109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:44.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10712", 'reported_seq': 10883, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.293956+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:40.293956+0000', 'last_peered': '2024-04-22T08:31:40.293956+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:40.293956+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38200530, 'num_objects': 9328, 'num_object_clones': 0, 'num_object_copies': 18656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3660, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10710, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5670}, {'shards': '0,2', 'objects': 3014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:44.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10719", 'reported_seq': 10988, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:40.326445+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:40.326445+0000', 'last_peered': '2024-04-22T08:31:40.326445+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:40.326445+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38025636, 'num_objects': 9287, 'num_object_clones': 0, 'num_object_copies': 18574, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3821, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10716, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5469}, {'shards': '0,2', 'objects': 3169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:44.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:45.437 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37476/297814 objects degraded (12.584%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":340,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":148907,"data_bytes":609873773,"bytes_used":2203021312,"bytes_avail":287707271168,"bytes_total":289910292480,"degraded_objects":37476,"degraded_total":297814,"degraded_ratio":0.1258369317762093,"misplaced_objects":1605,"misplaced_total":297814,"misplaced_ratio":0.0053892698127018879,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":112935,"recovering_keys_per_sec":0,"num_objects_recovered":520,"num_bytes_recovered":1355776,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":349},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:45.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:45.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:46.162 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:46.216 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-22T08:31:44.246037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609873773,"num_objects":148907,"num_object_clones":0,"num_object_copies":297814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":148907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144797,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2151388,"kb_used_data":1244412,"kb_used_omap":12,"kb_used_meta":906931,"kb_avail":280964132,"statfs":{"total":289910292480,"available":287707271168,"internally_reserved":0,"allocated":1274277888,"data_stored":1273579354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":928697454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,10,2,5,2,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17199104,"num_objects":-4199,"num_object_clones":0,"num_object_copies":-8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":-331,"num_objects_unfound":0,"num_objects_dirty":-4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":1355776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004891"},"pg_stats":[{"pgid":"2.f","version":"37'10774","reported_seq":10984,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.325556+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:40.325556+0000","last_peered":"2024-04-22T08:31:40.325556+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:40.325556+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":9394,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5673},{"shards":"0,2","objects":3104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10829","reported_seq":10944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.390153+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:40.390153+0000","last_peered":"2024-04-22T08:31:40.390153+0000","last_clean":"2024-04-22T08:31:40.390153+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:40.390153+0000","last_undegraded":"2024-04-22T08:31:40.390153+0000","last_fullsized":"2024-04-22T08:31:40.390153+0000","mapping_epoch":27,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514688,"num_objects":9403,"num_object_clones":0,"num_object_copies":18806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7885,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10600","reported_seq":10769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.372729+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:40.372729+0000","last_peered":"2024-04-22T08:31:40.372729+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:40.372729+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37941272,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7427,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3089},{"shards":"2","objects":5531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10761","reported_seq":10792,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.405369+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:40.405369+0000","last_peered":"2024-04-22T08:31:40.405369+0000","last_clean":"2024-04-22T08:31:40.405369+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:40.405369+0000","last_undegraded":"2024-04-22T08:31:40.405369+0000","last_fullsized":"2024-04-22T08:31:40.405369+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38309888,"num_objects":9353,"num_object_clones":0,"num_object_copies":18706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10663","reported_seq":10802,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.257620+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:40.257620+0000","last_peered":"2024-04-22T08:31:40.257620+0000","last_clean":"2024-04-22T08:31:40.257620+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:40.257620+0000","last_undegraded":"2024-04-22T08:31:40.257620+0000","last_fullsized":"2024-04-22T08:31:40.257620+0000","mapping_epoch":27,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37912576,"num_objects":9256,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10707","reported_seq":11020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.280590+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:40.280590+0000","last_peered":"2024-04-22T08:31:40.280590+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:40.280590+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":9340,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10737","reported_seq":18473,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:43.938811+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:43.938811+0000","last_peered":"2024-04-22T08:31:43.938811+0000","last_clean":"2024-04-22T08:31:43.938811+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:43.938811+0000","last_undegraded":"2024-04-22T08:31:43.938811+0000","last_fullsized":"2024-04-22T08:31:43.938811+0000","mapping_epoch":36,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37384192,"num_objects":9127,"num_object_clones":0,"num_object_copies":18254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10574","reported_seq":10718,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:40.335504+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:40.335504+0000","last_peered":"2024-04-22T08:31:40.335504+0000","last_clean":"2024-04-22T08:31:40.335504+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:40.335504+0000","last_undegraded":"2024-04-22T08:31:40.335504+0000","last_fullsized":"2024-04-22T08:31:40.335504+0000","mapping_epoch":27,"log_start":"28'500","ondisk_log_start":"28'500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":9232,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10861","reported_seq":11036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.447032+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:40.447032+0000","last_peered":"2024-04-22T08:31:40.447032+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:40.447032+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38339794,"num_objects":9362,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3170},{"shards":"2","objects":5501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10769","reported_seq":10985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.226021+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:40.226021+0000","last_peered":"2024-04-22T08:31:40.226021+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:40.226021+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38573266,"num_objects":9419,"num_object_clones":0,"num_object_copies":18838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5717},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10627","reported_seq":10846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.376745+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:40.376745+0000","last_peered":"2024-04-22T08:31:40.376745+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:40.376745+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":9302,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3090},{"shards":"2","objects":5588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10647","reported_seq":15104,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:40.452998+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:40.452998+0000","last_peered":"2024-04-22T08:31:40.452998+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:40.452998+0000","last_undegraded":"2024-04-22T08:31:40.452998+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2202,"num_bytes_recovered":9003008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1605},{"shards":"2","objects":7684}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10735","reported_seq":10943,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.302768+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:40.302768+0000","last_peered":"2024-04-22T08:31:40.302768+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:40.302768+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38531689,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5751},{"shards":"0,2","objects":3058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10607","reported_seq":10876,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.300082+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:40.300082+0000","last_peered":"2024-04-22T08:31:40.300082+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:40.300082+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'600","ondisk_log_start":"28'600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37449728,"num_objects":9143,"num_object_clones":0,"num_object_copies":18286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":3109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10712","reported_seq":10883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.293956+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:40.293956+0000","last_peered":"2024-04-22T08:31:40.293956+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:40.293956+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200530,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10719","reported_seq":10988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:40.326445+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:40.326445+0000","last_peered":"2024-04-22T08:31:40.326445+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:40.326445+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38025636,"num_objects":9287,"num_object_clones":0,"num_object_copies":18574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5469},{"shards":"0,2","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":609873773,"num_objects":148907,"num_object_clones":0,"num_object_copies":297814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37476,"num_objects_misplaced":1605,"num_objects_unfound":0,"num_objects_dirty":148907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144797,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6526,"num_bytes_recovered":24506985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273049088,"data_stored":1272967885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117052,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704044,"kb_used_data":419748,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93667796,"statfs":{"total":96636764160,"available":95915823104,"internally_reserved":0,"allocated":429821952,"data_stored":429606470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":3.0070000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":3.044}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.30199999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920815,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600668,"kb_used_data":316368,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93771172,"statfs":{"total":96636764160,"available":96021680128,"internally_reserved":0,"allocated":323960832,"data_stored":323682728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":291106508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.193},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.221}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313280,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":846676,"kb_used_data":508296,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93525164,"statfs":{"total":96636764160,"available":95769767936,"internally_reserved":0,"allocated":520495104,"data_stored":520290156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,1,1,0,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":5.5819999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":5.5369999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520085504,"data_stored":520082025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323637248,"data_stored":323567075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":429326336,"data_stored":429318785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:49.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:49.813 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:49.848 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-22T08:31:48.246920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602062701,"num_objects":147000,"num_object_clones":0,"num_object_copies":294000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37380,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":147000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146704,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6798,"num_bytes_recovered":25227881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2149400,"kb_used_data":1234232,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":280966120,"statfs":{"total":289910292480,"available":287709306880,"internally_reserved":0,"allocated":1263853568,"data_stored":1263155034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":937086062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,18,2,1,0,2,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16568320,"num_objects":-4045,"num_object_clones":0,"num_object_copies":-8090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":-339,"num_objects_unfound":0,"num_objects_dirty":-4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":548,"num_bytes_recovered":1388544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"2.f","version":"37'10894","reported_seq":11124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.280896+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:45.280896+0000","last_peered":"2024-04-22T08:31:45.280896+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:45.280896+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37986304,"num_objects":9274,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5563},{"shards":"0,2","objects":2984}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10948","reported_seq":11063,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.421228+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:45.421228+0000","last_peered":"2024-04-22T08:31:45.421228+0000","last_clean":"2024-04-22T08:31:45.421228+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:45.421228+0000","last_undegraded":"2024-04-22T08:31:45.421228+0000","last_fullsized":"2024-04-22T08:31:45.421228+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38027264,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8004,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10716","reported_seq":10893,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.355701+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:45.355701+0000","last_peered":"2024-04-22T08:31:45.355701+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:45.355701+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37466136,"num_objects":9148,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7543,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2973},{"shards":"2","objects":5419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10913","reported_seq":10944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.371544+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:45.371544+0000","last_peered":"2024-04-22T08:31:45.371544+0000","last_clean":"2024-04-22T08:31:45.371544+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:45.371544+0000","last_undegraded":"2024-04-22T08:31:45.371544+0000","last_fullsized":"2024-04-22T08:31:45.371544+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37687296,"num_objects":9201,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10913,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10794","reported_seq":10933,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.372139+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:45.372139+0000","last_peered":"2024-04-22T08:31:45.372139+0000","last_clean":"2024-04-22T08:31:45.372139+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:45.372139+0000","last_undegraded":"2024-04-22T08:31:45.372139+0000","last_fullsized":"2024-04-22T08:31:45.372139+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37376000,"num_objects":9125,"num_object_clones":0,"num_object_copies":18250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10861","reported_seq":11214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.374057+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:45.374057+0000","last_peered":"2024-04-22T08:31:45.374057+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:45.374057+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37625856,"num_objects":9186,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5402},{"shards":"0,2","objects":3044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10737","reported_seq":18473,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:43.938811+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:43.938811+0000","last_peered":"2024-04-22T08:31:43.938811+0000","last_clean":"2024-04-22T08:31:43.938811+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:43.938811+0000","last_undegraded":"2024-04-22T08:31:43.938811+0000","last_fullsized":"2024-04-22T08:31:43.938811+0000","mapping_epoch":36,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37384192,"num_objects":9127,"num_object_clones":0,"num_object_copies":18254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10712","reported_seq":10857,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.319221+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:45.319221+0000","last_peered":"2024-04-22T08:31:45.319221+0000","last_clean":"2024-04-22T08:31:45.319221+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:45.319221+0000","last_undegraded":"2024-04-22T08:31:45.319221+0000","last_fullsized":"2024-04-22T08:31:45.319221+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37249024,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10990","reported_seq":11173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.478426+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:45.478426+0000","last_peered":"2024-04-22T08:31:45.478426+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:45.478426+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811410,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10930,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3041},{"shards":"2","objects":5375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10880","reported_seq":11122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.371109+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:45.371109+0000","last_peered":"2024-04-22T08:31:45.371109+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:45.371109+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38118610,"num_objects":9308,"num_object_clones":0,"num_object_copies":18616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5619},{"shards":"0,2","objects":2988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10745","reported_seq":10968,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.464283+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:45.464283+0000","last_peered":"2024-04-22T08:31:45.464283+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:45.464283+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":9184,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2972},{"shards":"2","objects":5473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10759","reported_seq":15568,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:45.412145+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:45.412145+0000","last_peered":"2024-04-22T08:31:45.412145+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:45.412145+0000","last_undegraded":"2024-04-22T08:31:45.412145+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582034,"num_objects":9177,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":9177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2378,"num_bytes_recovered":9723904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1429},{"shards":"2","objects":7748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10859","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.231771+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:45.231771+0000","last_peered":"2024-04-22T08:31:45.231771+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:45.231771+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38023785,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5635},{"shards":"0,2","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10726","reported_seq":11019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.339398+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:45.339398+0000","last_peered":"2024-04-22T08:31:45.339398+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:45.339398+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":9024,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5246},{"shards":"0,2","objects":2990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10849","reported_seq":11044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.357566+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:45.357566+0000","last_peered":"2024-04-22T08:31:45.357566+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:45.357566+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639378,"num_objects":9191,"num_object_clones":0,"num_object_copies":18382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5545},{"shards":"0,2","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10846","reported_seq":11138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.289132+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:45.289132+0000","last_peered":"2024-04-22T08:31:45.289132+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:45.289132+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37505444,"num_objects":9160,"num_object_clones":0,"num_object_copies":18320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":602062701,"num_objects":147000,"num_object_clones":0,"num_object_copies":294000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37380,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":147000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146704,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6798,"num_bytes_recovered":25227881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1262522368,"data_stored":1262441165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117053,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704200,"kb_used_data":415808,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93667640,"statfs":{"total":96636764160,"available":95915663360,"internally_reserved":0,"allocated":425787392,"data_stored":425571910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295300815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,1,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":6.673},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.311}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.223}]}]},{"osd":1,"up_from":33,"seq":141733920815,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600668,"kb_used_data":316368,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93771172,"statfs":{"total":96636764160,"available":96021680128,"internally_reserved":0,"allocated":323960832,"data_stored":323682728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":291106508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.193},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.221}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313281,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844532,"kb_used_data":502056,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93527308,"statfs":{"total":96636764160,"available":95771963392,"internally_reserved":0,"allocated":514105344,"data_stored":513900396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,18,1,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":513699840,"data_stored":513696361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323637248,"data_stored":323567075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":425185280,"data_stored":425177729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:49.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:50.569 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:50.570 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:50.633 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-04-22T08:31:50.247366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601509741,"num_objects":146865,"num_object_clones":0,"num_object_copies":293730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37380,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":146865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146839,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6798,"num_bytes_recovered":25227881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2153044,"kb_used_data":1233784,"kb_used_omap":12,"kb_used_meta":919219,"kb_avail":280962476,"statfs":{"total":289910292480,"available":287705575424,"internally_reserved":0,"allocated":1263394816,"data_stored":1262696282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":941280366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,21,3,1,0,2,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17121280,"num_objects":-4180,"num_object_clones":0,"num_object_copies":-8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":-339,"num_objects_unfound":0,"num_objects_dirty":-4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":548,"num_bytes_recovered":1388544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002869"},"pg_stats":[{"pgid":"2.f","version":"37'10894","reported_seq":11124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.280896+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:45.280896+0000","last_peered":"2024-04-22T08:31:45.280896+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:45.280896+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37986304,"num_objects":9274,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5563},{"shards":"0,2","objects":2984}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'10948","reported_seq":11063,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.421228+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:45.421228+0000","last_peered":"2024-04-22T08:31:45.421228+0000","last_clean":"2024-04-22T08:31:45.421228+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:45.421228+0000","last_undegraded":"2024-04-22T08:31:45.421228+0000","last_fullsized":"2024-04-22T08:31:45.421228+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38027264,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8004,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10716","reported_seq":10893,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.355701+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:45.355701+0000","last_peered":"2024-04-22T08:31:45.355701+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:45.355701+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37466136,"num_objects":9148,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7543,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2973},{"shards":"2","objects":5419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'10913","reported_seq":10944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.371544+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:45.371544+0000","last_peered":"2024-04-22T08:31:45.371544+0000","last_clean":"2024-04-22T08:31:45.371544+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:45.371544+0000","last_undegraded":"2024-04-22T08:31:45.371544+0000","last_fullsized":"2024-04-22T08:31:45.371544+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37687296,"num_objects":9201,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10913,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10794","reported_seq":10933,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.372139+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:45.372139+0000","last_peered":"2024-04-22T08:31:45.372139+0000","last_clean":"2024-04-22T08:31:45.372139+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:45.372139+0000","last_undegraded":"2024-04-22T08:31:45.372139+0000","last_fullsized":"2024-04-22T08:31:45.372139+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37376000,"num_objects":9125,"num_object_clones":0,"num_object_copies":18250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10861","reported_seq":11214,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.374057+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:45.374057+0000","last_peered":"2024-04-22T08:31:45.374057+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:45.374057+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37625856,"num_objects":9186,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5402},{"shards":"0,2","objects":3044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10872","reported_seq":18608,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:49.078026+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:49.078026+0000","last_peered":"2024-04-22T08:31:49.078026+0000","last_clean":"2024-04-22T08:31:49.078026+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:49.078026+0000","last_undegraded":"2024-04-22T08:31:49.078026+0000","last_fullsized":"2024-04-22T08:31:49.078026+0000","mapping_epoch":36,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36831232,"num_objects":8992,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10712","reported_seq":10857,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:45.319221+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:45.319221+0000","last_peered":"2024-04-22T08:31:45.319221+0000","last_clean":"2024-04-22T08:31:45.319221+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:45.319221+0000","last_undegraded":"2024-04-22T08:31:45.319221+0000","last_fullsized":"2024-04-22T08:31:45.319221+0000","mapping_epoch":27,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37249024,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'10990","reported_seq":11173,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.478426+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:45.478426+0000","last_peered":"2024-04-22T08:31:45.478426+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:45.478426+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811410,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10930,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3041},{"shards":"2","objects":5375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10880","reported_seq":11122,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.371109+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:45.371109+0000","last_peered":"2024-04-22T08:31:45.371109+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:45.371109+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38118610,"num_objects":9308,"num_object_clones":0,"num_object_copies":18616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5619},{"shards":"0,2","objects":2988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10745","reported_seq":10968,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.464283+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:45.464283+0000","last_peered":"2024-04-22T08:31:45.464283+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:45.464283+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":9184,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2972},{"shards":"2","objects":5473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10759","reported_seq":15568,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:45.412145+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:45.412145+0000","last_peered":"2024-04-22T08:31:45.412145+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:45.412145+0000","last_undegraded":"2024-04-22T08:31:45.412145+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582034,"num_objects":9177,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":9177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2378,"num_bytes_recovered":9723904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1429},{"shards":"2","objects":7748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10859","reported_seq":11083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.231771+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:45.231771+0000","last_peered":"2024-04-22T08:31:45.231771+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:45.231771+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38023785,"num_objects":9284,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5635},{"shards":"0,2","objects":2934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10726","reported_seq":11019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.339398+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:45.339398+0000","last_peered":"2024-04-22T08:31:45.339398+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:45.339398+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'700","ondisk_log_start":"28'700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":9024,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5246},{"shards":"0,2","objects":2990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10849","reported_seq":11044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.357566+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:45.357566+0000","last_peered":"2024-04-22T08:31:45.357566+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:45.357566+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639378,"num_objects":9191,"num_object_clones":0,"num_object_copies":18382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5545},{"shards":"0,2","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10846","reported_seq":11138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:45.289132+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:45.289132+0000","last_peered":"2024-04-22T08:31:45.289132+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:45.289132+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37505444,"num_objects":9160,"num_object_clones":0,"num_object_copies":18320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":601509741,"num_objects":146865,"num_object_clones":0,"num_object_copies":293730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37380,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":146865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146839,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6798,"num_bytes_recovered":25227881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1262047232,"data_stored":1261966029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117053,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704200,"kb_used_data":415808,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93667640,"statfs":{"total":96636764160,"available":95915663360,"internally_reserved":0,"allocated":425787392,"data_stored":425571910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295300815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,1,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":6.673},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.311}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.223}]}]},{"osd":1,"up_from":33,"seq":141733920816,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604312,"kb_used_data":315920,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93767528,"statfs":{"total":96636764160,"available":96017948672,"internally_reserved":0,"allocated":323502080,"data_stored":323223976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.379},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.438}]}]},{"osd":0,"up_from":21,"seq":90194313281,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844532,"kb_used_data":502056,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93527308,"statfs":{"total":96636764160,"available":95771963392,"internally_reserved":0,"allocated":514105344,"data_stored":513900396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,18,1,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":513699840,"data_stored":513696361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323162112,"data_stored":323091939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":425185280,"data_stored":425177729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'10894", 'reported_seq': 11124, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.280896+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:45.280896+0000', 'last_peered': '2024-04-22T08:31:45.280896+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:45.280896+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37986304, 'num_objects': 9274, 'num_object_clones': 0, 'num_object_copies': 18548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3711, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10894, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5563}, {'shards': '0,2', 'objects': 2984}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10716", 'reported_seq': 10893, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.355701+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:45.355701+0000', 'last_peered': '2024-04-22T08:31:45.355701+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:45.355701+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37466136, 'num_objects': 9148, 'num_object_clones': 0, 'num_object_copies': 18296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3730, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9148, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7543, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2973}, {'shards': '2', 'objects': 5419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'10861", 'reported_seq': 11214, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.374057+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:45.374057+0000', 'last_peered': '2024-04-22T08:31:45.374057+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:45.374057+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37625856, 'num_objects': 9186, 'num_object_clones': 0, 'num_object_copies': 18372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3786, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7730, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 96, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5402}, {'shards': '0,2', 'objects': 3044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'10990", 'reported_seq': 11173, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.478426+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:45.478426+0000', 'last_peered': '2024-04-22T08:31:45.478426+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:45.478426+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37811410, 'num_objects': 9233, 'num_object_clones': 0, 'num_object_copies': 18466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3860, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10930, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3041}, {'shards': '2', 'objects': 5375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'10880", 'reported_seq': 11122, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.371109+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:45.371109+0000', 'last_peered': '2024-04-22T08:31:45.371109+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:45.371109+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38118610, 'num_objects': 9308, 'num_object_clones': 0, 'num_object_copies': 18616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10878, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5619}, {'shards': '0,2', 'objects': 2988}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'10745", 'reported_seq': 10968, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.464283+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:45.464283+0000', 'last_peered': '2024-04-22T08:31:45.464283+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:45.464283+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37617664, 'num_objects': 9184, 'num_object_clones': 0, 'num_object_copies': 18368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3713, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7199, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2972}, {'shards': '2', 'objects': 5473}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'10759", 'reported_seq': 15568, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:45.412145+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:45.412145+0000', 'last_peered': '2024-04-22T08:31:45.412145+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:45.412145+0000', 'last_undegraded': '2024-04-22T08:31:45.412145+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37582034, 'num_objects': 9177, 'num_object_clones': 0, 'num_object_copies': 18354, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1429, 'num_objects_unfound': 0, 'num_objects_dirty': 9177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10757, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2378, 'num_bytes_recovered': 9723904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1429}, {'shards': '2', 'objects': 7748}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'10859", 'reported_seq': 11083, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.231771+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:45.231771+0000', 'last_peered': '2024-04-22T08:31:45.231771+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:45.231771+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38023785, 'num_objects': 9284, 'num_object_clones': 0, 'num_object_copies': 18568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3650, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10858, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5635}, {'shards': '0,2', 'objects': 2934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'10726", 'reported_seq': 11019, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.339398+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:45.339398+0000', 'last_peered': '2024-04-22T08:31:45.339398+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:45.339398+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36962304, 'num_objects': 9024, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3781, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6957, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5246}, {'shards': '0,2', 'objects': 2990}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'10849", 'reported_seq': 11044, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.357566+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:45.357566+0000', 'last_peered': '2024-04-22T08:31:45.357566+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:45.357566+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37639378, 'num_objects': 9191, 'num_object_clones': 0, 'num_object_copies': 18382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3648, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10847, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5545}, {'shards': '0,2', 'objects': 2877}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:50.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'10846", 'reported_seq': 11138, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:45.289132+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:45.289132+0000', 'last_peered': '2024-04-22T08:31:45.289132+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:45.289132+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37505444, 'num_objects': 9160, 'num_object_clones': 0, 'num_object_copies': 18320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3810, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10843, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5353}, {'shards': '0,2', 'objects': 3042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:50.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:51.480 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:51.480 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37380/293730 objects degraded (12.726%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":146865,"data_bytes":601509741,"bytes_used":2204717056,"bytes_avail":287705575424,"bytes_total":289910292480,"degraded_objects":37380,"degraded_total":293730,"degraded_ratio":0.12725972832192831,"misplaced_objects":1429,"misplaced_total":293730,"misplaced_ratio":0.0048650120859292552,"recovering_objects_per_sec":45,"recovering_bytes_per_sec":115684,"recovering_keys_per_sec":0,"num_objects_recovered":548,"num_bytes_recovered":1388544,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":348},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:51.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:51.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:52.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:52.281 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:52.335 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-04-22T08:31:52.247870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593715053,"num_objects":144962,"num_object_clones":0,"num_object_copies":289924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37307,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":144962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148742,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7038,"num_bytes_recovered":25911913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160867,"ondisk_log_size":160867,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2151140,"kb_used_data":1223684,"kb_used_omap":12,"kb_used_meta":927411,"kb_avail":280964380,"statfs":{"total":289910292480,"available":287707525120,"internally_reserved":0,"allocated":1253052416,"data_stored":1252353882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":949668974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,6,5,1,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24334336,"num_objects":-5941,"num_object_clones":0,"num_object_copies":-11882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":-506,"num_objects_unfound":0,"num_objects_dirty":-5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":788,"num_bytes_recovered":2072576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.f","version":"37'11019","reported_seq":11263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.221529+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:50.221529+0000","last_peered":"2024-04-22T08:31:50.221529+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:50.221529+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37474304,"num_objects":9149,"num_object_clones":0,"num_object_copies":18298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5445},{"shards":"0,2","objects":2859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11083","reported_seq":11198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.512042+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:50.512042+0000","last_peered":"2024-04-22T08:31:50.512042+0000","last_clean":"2024-04-22T08:31:50.512042+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:50.512042+0000","last_undegraded":"2024-04-22T08:31:50.512042+0000","last_fullsized":"2024-04-22T08:31:50.512042+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37474304,"num_objects":9149,"num_object_clones":0,"num_object_copies":18298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10827","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.444927+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:50.444927+0000","last_peered":"2024-04-22T08:31:50.444927+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:50.444927+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37011480,"num_objects":9037,"num_object_clones":0,"num_object_copies":18074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7654,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2862},{"shards":"2","objects":5314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11020","reported_seq":11051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.452110+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:50.452110+0000","last_peered":"2024-04-22T08:31:50.452110+0000","last_clean":"2024-04-22T08:31:50.452110+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:50.452110+0000","last_undegraded":"2024-04-22T08:31:50.452110+0000","last_fullsized":"2024-04-22T08:31:50.452110+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37249024,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10913","reported_seq":11052,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.354040+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:50.354040+0000","last_peered":"2024-04-22T08:31:50.354040+0000","last_clean":"2024-04-22T08:31:50.354040+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:50.354040+0000","last_undegraded":"2024-04-22T08:31:50.354040+0000","last_fullsized":"2024-04-22T08:31:50.354040+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36888576,"num_objects":9006,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8005,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10969","reported_seq":11342,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.326193+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:50.326193+0000","last_peered":"2024-04-22T08:31:50.326193+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:50.326193+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37183488,"num_objects":9078,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5304},{"shards":"0,2","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'10872","reported_seq":18608,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:49.078026+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:49.078026+0000","last_peered":"2024-04-22T08:31:49.078026+0000","last_clean":"2024-04-22T08:31:49.078026+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:49.078026+0000","last_undegraded":"2024-04-22T08:31:49.078026+0000","last_fullsized":"2024-04-22T08:31:49.078026+0000","mapping_epoch":36,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36831232,"num_objects":8992,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10839","reported_seq":10984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.379624+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:50.379624+0000","last_peered":"2024-04-22T08:31:50.379624+0000","last_clean":"2024-04-22T08:31:50.379624+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:50.379624+0000","last_undegraded":"2024-04-22T08:31:50.379624+0000","last_fullsized":"2024-04-22T08:31:50.379624+0000","mapping_epoch":27,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36728832,"num_objects":8967,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.165731+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:09.165731+0000","last_peered":"2024-04-22T08:30:09.165731+0000","last_clean":"2024-04-22T08:30:09.165731+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:09.165731+0000","last_undegraded":"2024-04-22T08:30:09.165731+0000","last_fullsized":"2024-04-22T08:30:09.165731+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11127","reported_seq":11321,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.490164+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:50.490164+0000","last_peered":"2024-04-22T08:31:50.490164+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:50.490164+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37250258,"num_objects":9096,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2904},{"shards":"2","objects":5244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10995","reported_seq":11253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.296847+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:50.296847+0000","last_peered":"2024-04-22T08:31:50.296847+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:50.296847+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37647570,"num_objects":9193,"num_object_clones":0,"num_object_copies":18386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5512},{"shards":"0,2","objects":2873}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10881","reported_seq":11130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.511964+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:50.511964+0000","last_peered":"2024-04-22T08:31:50.511964+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:50.511964+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37060608,"num_objects":9048,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7335,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2836},{"shards":"2","objects":5349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10913","reported_seq":16056,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:50.533171+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:50.533171+0000","last_peered":"2024-04-22T08:31:50.533171+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:50.533171+0000","last_undegraded":"2024-04-22T08:31:50.533171+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951250,"num_objects":9023,"num_object_clones":0,"num_object_copies":18046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":9023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2545,"num_bytes_recovered":10407936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1262},{"shards":"2","objects":7761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10985","reported_seq":11225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.309061+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:50.309061+0000","last_peered":"2024-04-22T08:31:50.309061+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:50.309061+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37507689,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5518},{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10879","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.247979+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:50.247979+0000","last_peered":"2024-04-22T08:31:50.247979+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:50.247979+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36335616,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5098},{"shards":"0,2","objects":2837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10978","reported_seq":11183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.361651+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:50.361651+0000","last_peered":"2024-04-22T08:31:50.361651+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:50.361651+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37110994,"num_objects":9062,"num_object_clones":0,"num_object_copies":18124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5421},{"shards":"0,2","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10967","reported_seq":11269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.357318+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:50.357318+0000","last_peered":"2024-04-22T08:31:50.357318+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:50.357318+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37009828,"num_objects":9039,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10964,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5237},{"shards":"0,2","objects":2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":593715053,"num_objects":144962,"num_object_clones":0,"num_object_copies":289924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37307,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":144962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148742,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7038,"num_bytes_recovered":25911913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1251655680,"data_stored":1251574477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160867,"ondisk_log_size":160867,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117054,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704212,"kb_used_data":411720,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93667628,"statfs":{"total":96636764160,"available":95915651072,"internally_reserved":0,"allocated":421601280,"data_stored":421385798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299495119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.62}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.7270000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":3.9700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920816,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604312,"kb_used_data":315920,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93767528,"statfs":{"total":96636764160,"available":96017948672,"internally_reserved":0,"allocated":323502080,"data_stored":323223976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.379},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.438}]}]},{"osd":0,"up_from":21,"seq":90194313282,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":842616,"kb_used_data":496044,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93529224,"statfs":{"total":96636764160,"available":95773925376,"internally_reserved":0,"allocated":507949056,"data_stored":507744108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.245},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":507547648,"data_stored":507544169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323162112,"data_stored":323091939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":420945920,"data_stored":420938369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:55.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:55.893 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:55.894 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:55.977 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-04-22T08:31:54.248366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593182573,"num_objects":144832,"num_object_clones":0,"num_object_copies":289664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37307,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":144832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148872,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7038,"num_bytes_recovered":25911913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2142688,"kb_used_data":1223424,"kb_used_omap":12,"kb_used_meta":919219,"kb_avail":280972832,"statfs":{"total":289910292480,"available":287716179968,"internally_reserved":0,"allocated":1252786176,"data_stored":1252087642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":941280366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4,4,1,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17248256,"num_objects":-4211,"num_object_clones":0,"num_object_copies":-8422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":-343,"num_objects_unfound":0,"num_objects_dirty":-4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":1404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002875"},"pg_stats":[{"pgid":"2.f","version":"37'11019","reported_seq":11263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.221529+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:50.221529+0000","last_peered":"2024-04-22T08:31:50.221529+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:50.221529+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37474304,"num_objects":9149,"num_object_clones":0,"num_object_copies":18298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5445},{"shards":"0,2","objects":2859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11083","reported_seq":11198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.512042+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:50.512042+0000","last_peered":"2024-04-22T08:31:50.512042+0000","last_clean":"2024-04-22T08:31:50.512042+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:50.512042+0000","last_undegraded":"2024-04-22T08:31:50.512042+0000","last_fullsized":"2024-04-22T08:31:50.512042+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37474304,"num_objects":9149,"num_object_clones":0,"num_object_copies":18298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10827","reported_seq":11016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.444927+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:50.444927+0000","last_peered":"2024-04-22T08:31:50.444927+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:50.444927+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37011480,"num_objects":9037,"num_object_clones":0,"num_object_copies":18074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7654,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2862},{"shards":"2","objects":5314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11020","reported_seq":11051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.452110+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:50.452110+0000","last_peered":"2024-04-22T08:31:50.452110+0000","last_clean":"2024-04-22T08:31:50.452110+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:50.452110+0000","last_undegraded":"2024-04-22T08:31:50.452110+0000","last_fullsized":"2024-04-22T08:31:50.452110+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37249024,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'10913","reported_seq":11052,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.354040+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:50.354040+0000","last_peered":"2024-04-22T08:31:50.354040+0000","last_clean":"2024-04-22T08:31:50.354040+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:50.354040+0000","last_undegraded":"2024-04-22T08:31:50.354040+0000","last_fullsized":"2024-04-22T08:31:50.354040+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36888576,"num_objects":9006,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8005,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'10969","reported_seq":11342,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.326193+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:50.326193+0000","last_peered":"2024-04-22T08:31:50.326193+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:50.326193+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37183488,"num_objects":9078,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5304},{"shards":"0,2","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11002","reported_seq":18737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:54.015074+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:54.015074+0000","last_peered":"2024-04-22T08:31:54.015074+0000","last_clean":"2024-04-22T08:31:54.015074+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:54.015074+0000","last_undegraded":"2024-04-22T08:31:54.015074+0000","last_fullsized":"2024-04-22T08:31:54.015074+0000","mapping_epoch":36,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36298752,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10839","reported_seq":10984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:50.379624+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:50.379624+0000","last_peered":"2024-04-22T08:31:50.379624+0000","last_clean":"2024-04-22T08:31:50.379624+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:50.379624+0000","last_undegraded":"2024-04-22T08:31:50.379624+0000","last_fullsized":"2024-04-22T08:31:50.379624+0000","mapping_epoch":27,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36728832,"num_objects":8967,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11127","reported_seq":11321,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.490164+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:50.490164+0000","last_peered":"2024-04-22T08:31:50.490164+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:50.490164+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37250258,"num_objects":9096,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2904},{"shards":"2","objects":5244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'10995","reported_seq":11253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.296847+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:50.296847+0000","last_peered":"2024-04-22T08:31:50.296847+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:50.296847+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37647570,"num_objects":9193,"num_object_clones":0,"num_object_copies":18386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5512},{"shards":"0,2","objects":2873}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'10881","reported_seq":11130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.511964+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:50.511964+0000","last_peered":"2024-04-22T08:31:50.511964+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:50.511964+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37060608,"num_objects":9048,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7335,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2836},{"shards":"2","objects":5349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'10913","reported_seq":16056,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:50.533171+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:50.533171+0000","last_peered":"2024-04-22T08:31:50.533171+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:50.533171+0000","last_undegraded":"2024-04-22T08:31:50.533171+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951250,"num_objects":9023,"num_object_clones":0,"num_object_copies":18046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":9023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2545,"num_bytes_recovered":10407936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1262},{"shards":"2","objects":7761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'10985","reported_seq":11225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.309061+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:50.309061+0000","last_peered":"2024-04-22T08:31:50.309061+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:50.309061+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37507689,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5518},{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'10879","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.247979+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:50.247979+0000","last_peered":"2024-04-22T08:31:50.247979+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:50.247979+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'800","ondisk_log_start":"28'800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36335616,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5098},{"shards":"0,2","objects":2837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'10978","reported_seq":11183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.361651+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:50.361651+0000","last_peered":"2024-04-22T08:31:50.361651+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:50.361651+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37110994,"num_objects":9062,"num_object_clones":0,"num_object_copies":18124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5421},{"shards":"0,2","objects":2748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'10967","reported_seq":11269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:50.357318+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:50.357318+0000","last_peered":"2024-04-22T08:31:50.357318+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:50.357318+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37009828,"num_objects":9039,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10964,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5237},{"shards":"0,2","objects":2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":593182573,"num_objects":144832,"num_object_clones":0,"num_object_copies":289664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37307,"num_objects_misplaced":1262,"num_objects_unfound":0,"num_objects_dirty":144832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148872,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7038,"num_bytes_recovered":25911913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1251405824,"data_stored":1251324621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117054,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704212,"kb_used_data":411720,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93667628,"statfs":{"total":96636764160,"available":95915651072,"internally_reserved":0,"allocated":421601280,"data_stored":421385798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299495119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.62}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.7270000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":3.9700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920817,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595860,"kb_used_data":315660,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93775980,"statfs":{"total":96636764160,"available":96026603520,"internally_reserved":0,"allocated":323235840,"data_stored":322957736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313282,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":842616,"kb_used_data":496044,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93529224,"statfs":{"total":96636764160,"available":95773925376,"internally_reserved":0,"allocated":507949056,"data_stored":507744108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.245},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":507547648,"data_stored":507544169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322912256,"data_stored":322842083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":420945920,"data_stored":420938369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:55.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:56.909 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:56.911 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:56.971 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-22T08:31:56.248853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585473901,"num_objects":142950,"num_object_clones":0,"num_object_copies":285900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37187,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":142950,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150754,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7366,"num_bytes_recovered":26763881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2128460,"kb_used_data":1213296,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":280987060,"statfs":{"total":289910292480,"available":287730749440,"internally_reserved":0,"allocated":1242415104,"data_stored":1241716570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":937086062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,8,6,0,0,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24399872,"num_objects":-5957,"num_object_clones":0,"num_object_copies":-11914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-289,"num_objects_misplaced":-551,"num_objects_unfound":0,"num_objects_dirty":-5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":840,"num_bytes_recovered":2256896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002816"},"pg_stats":[{"pgid":"2.f","version":"37'11166","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.360764+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:55.360764+0000","last_peered":"2024-04-22T08:31:55.360764+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:55.360764+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36872192,"num_objects":9002,"num_object_clones":0,"num_object_copies":18004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11166,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5317},{"shards":"0,2","objects":2712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11209","reported_seq":11324,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.383045+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:55.383045+0000","last_peered":"2024-04-22T08:31:55.383045+0000","last_clean":"2024-04-22T08:31:55.383045+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:55.383045+0000","last_undegraded":"2024-04-22T08:31:55.383045+0000","last_fullsized":"2024-04-22T08:31:55.383045+0000","mapping_epoch":27,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36958208,"num_objects":9023,"num_object_clones":0,"num_object_copies":18046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8265,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10951","reported_seq":11150,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.514859+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:55.514859+0000","last_peered":"2024-04-22T08:31:55.514859+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:55.514859+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503576,"num_objects":8913,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7778,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2738},{"shards":"2","objects":5195}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11155","reported_seq":11186,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.521717+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:55.521717+0000","last_peered":"2024-04-22T08:31:55.521717+0000","last_clean":"2024-04-22T08:31:55.521717+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:55.521717+0000","last_undegraded":"2024-04-22T08:31:55.521717+0000","last_fullsized":"2024-04-22T08:31:55.521717+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36696064,"num_objects":8959,"num_object_clones":0,"num_object_copies":17918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11155,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11029","reported_seq":11168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.371143+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:55.371143+0000","last_peered":"2024-04-22T08:31:55.371143+0000","last_clean":"2024-04-22T08:31:55.371143+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:55.371143+0000","last_undegraded":"2024-04-22T08:31:55.371143+0000","last_fullsized":"2024-04-22T08:31:55.371143+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11099","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.391869+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:55.391869+0000","last_peered":"2024-04-22T08:31:55.391869+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:55.391869+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36651008,"num_objects":8948,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5187},{"shards":"0,2","objects":2806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11002","reported_seq":18737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:54.015074+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:54.015074+0000","last_peered":"2024-04-22T08:31:54.015074+0000","last_clean":"2024-04-22T08:31:54.015074+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:54.015074+0000","last_undegraded":"2024-04-22T08:31:54.015074+0000","last_fullsized":"2024-04-22T08:31:54.015074+0000","mapping_epoch":36,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36298752,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10958","reported_seq":11103,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.302869+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:55.302869+0000","last_peered":"2024-04-22T08:31:55.302869+0000","last_clean":"2024-04-22T08:31:55.302869+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:55.302869+0000","last_undegraded":"2024-04-22T08:31:55.302869+0000","last_fullsized":"2024-04-22T08:31:55.302869+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":8848,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11253","reported_seq":11457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.556777+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:55.556777+0000","last_peered":"2024-04-22T08:31:55.556777+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:55.556777+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36734162,"num_objects":8970,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2778},{"shards":"2","objects":5122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11113","reported_seq":11409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.359764+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:55.359764+0000","last_peered":"2024-04-22T08:31:55.359764+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:55.359764+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37164242,"num_objects":9075,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5413},{"shards":"0,2","objects":2755}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11019","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.543469+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:55.543469+0000","last_peered":"2024-04-22T08:31:55.543469+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:55.543469+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":8910,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2698},{"shards":"2","objects":5221}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11027","reported_seq":16586,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:55.545488+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:55.545488+0000","last_peered":"2024-04-22T08:31:55.545488+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:55.545488+0000","last_undegraded":"2024-04-22T08:31:55.545488+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36484306,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11025,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2753,"num_bytes_recovered":11259904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1054},{"shards":"2","objects":7855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11107","reported_seq":11393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.439746+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:55.439746+0000","last_peered":"2024-04-22T08:31:55.439746+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:55.439746+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37007977,"num_objects":9036,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5418},{"shards":"0,2","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11006","reported_seq":11335,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.368416+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:55.368416+0000","last_peered":"2024-04-22T08:31:55.368416+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:55.368416+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":8744,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4984},{"shards":"0,2","objects":2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11096","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.302847+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:55.302847+0000","last_peered":"2024-04-22T08:31:55.302847+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:55.302847+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36627666,"num_objects":8944,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5310},{"shards":"0,2","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11089","reported_seq":11407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.317784+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:55.317784+0000","last_peered":"2024-04-22T08:31:55.317784+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:55.317784+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36510116,"num_objects":8917,"num_object_clones":0,"num_object_copies":17834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5123},{"shards":"0,2","objects":2799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":585473901,"num_objects":142950,"num_object_clones":0,"num_object_copies":285900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37187,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":142950,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150754,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7366,"num_bytes_recovered":26763881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241169920,"data_stored":1241088717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117055,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704268,"kb_used_data":407684,"kb_used_omap":4,"kb_used_meta":296571,"kb_avail":93667572,"statfs":{"total":96636764160,"available":95915593728,"internally_reserved":0,"allocated":417468416,"data_stored":417252934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":303689423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,3,0,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":5.4509999999999996}]}]},{"osd":1,"up_from":33,"seq":141733920817,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595860,"kb_used_data":315660,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93775980,"statfs":{"total":96636764160,"available":96026603520,"internally_reserved":0,"allocated":323235840,"data_stored":322957736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313283,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":828332,"kb_used_data":489952,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93543508,"statfs":{"total":96636764160,"available":95788552192,"internally_reserved":0,"allocated":501710848,"data_stored":501505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.222},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":501366784,"data_stored":501363305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322912256,"data_stored":322842083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416890880,"data_stored":416883329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:31:56.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:31:56.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'11166", 'reported_seq': 11448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.360764+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:31:55.360764+0000', 'last_peered': '2024-04-22T08:31:55.360764+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:31:55.360764+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36872192, 'num_objects': 9002, 'num_object_clones': 0, 'num_object_copies': 18004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3685, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11166, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 109, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5317}, {'shards': '0,2', 'objects': 2712}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:31:56.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'10951", 'reported_seq': 11150, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.514859+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:31:55.514859+0000', 'last_peered': '2024-04-22T08:31:55.514859+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:31:55.514859+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36503576, 'num_objects': 8913, 'num_object_clones': 0, 'num_object_copies': 17826, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3719, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8913, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7778, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2738}, {'shards': '2', 'objects': 5195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:56.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'11099", 'reported_seq': 11498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.391869+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:31:55.391869+0000', 'last_peered': '2024-04-22T08:31:55.391869+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:31:55.391869+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36651008, 'num_objects': 8948, 'num_object_clones': 0, 'num_object_copies': 17896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3763, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7968, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5187}, {'shards': '0,2', 'objects': 2806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'11253", 'reported_seq': 11457, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.556777+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:31:55.556777+0000', 'last_peered': '2024-04-22T08:31:55.556777+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:31:55.556777+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36734162, 'num_objects': 8970, 'num_object_clones': 0, 'num_object_copies': 17940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3850, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2778}, {'shards': '2', 'objects': 5122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'11113", 'reported_seq': 11409, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.359764+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:31:55.359764+0000', 'last_peered': '2024-04-22T08:31:55.359764+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:31:55.359764+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37164242, 'num_objects': 9075, 'num_object_clones': 0, 'num_object_copies': 18150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3664, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11111, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 112, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5413}, {'shards': '0,2', 'objects': 2755}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'11019", 'reported_seq': 11287, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.543469+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:31:55.543469+0000', 'last_peered': '2024-04-22T08:31:55.543469+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:31:55.543469+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36495360, 'num_objects': 8910, 'num_object_clones': 0, 'num_object_copies': 17820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7473, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2698}, {'shards': '2', 'objects': 5221}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'11027", 'reported_seq': 16586, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:31:55.545488+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:31:55.545488+0000', 'last_peered': '2024-04-22T08:31:55.545488+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:31:55.545488+0000', 'last_undegraded': '2024-04-22T08:31:55.545488+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36484306, 'num_objects': 8909, 'num_object_clones': 0, 'num_object_copies': 17818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1054, 'num_objects_unfound': 0, 'num_objects_dirty': 8909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11025, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2753, 'num_bytes_recovered': 11259904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1054}, {'shards': '2', 'objects': 7855}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:31:56.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'11107", 'reported_seq': 11393, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.439746+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:31:55.439746+0000', 'last_peered': '2024-04-22T08:31:55.439746+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:31:55.439746+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37007977, 'num_objects': 9036, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3619, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11106, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 105, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5418}, {'shards': '0,2', 'objects': 2686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'11006", 'reported_seq': 11335, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.368416+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:31:55.368416+0000', 'last_peered': '2024-04-22T08:31:55.368416+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:31:55.368416+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35815424, 'num_objects': 8744, 'num_object_clones': 0, 'num_object_copies': 17488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3763, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7237, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4984}, {'shards': '0,2', 'objects': 2710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'11096", 'reported_seq': 11315, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.302847+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:31:55.302847+0000', 'last_peered': '2024-04-22T08:31:55.302847+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:31:55.302847+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36627666, 'num_objects': 8944, 'num_object_clones': 0, 'num_object_copies': 17888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3636, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11094, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5310}, {'shards': '0,2', 'objects': 2630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:31:56.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'11089", 'reported_seq': 11407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:31:55.317784+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:31:55.317784+0000', 'last_peered': '2024-04-22T08:31:55.317784+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:31:55.317784+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36510116, 'num_objects': 8917, 'num_object_clones': 0, 'num_object_copies': 17834, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11086, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 92, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5123}, {'shards': '0,2', 'objects': 2799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:31:56.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:31:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:57.846 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37187/285900 objects degraded (13.007%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":142950,"data_bytes":585473901,"bytes_used":2179543040,"bytes_avail":287730749440,"bytes_total":289910292480,"degraded_objects":37187,"degraded_total":285900,"degraded_ratio":0.13006995452955578,"misplaced_objects":1054,"misplaced_total":285900,"misplaced_ratio":0.0036866037075900665,"recovering_objects_per_sec":69,"recovering_bytes_per_sec":188030,"recovering_keys_per_sec":0,"num_objects_recovered":840,"num_bytes_recovered":2256896,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":496},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:31:57.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:31:57.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:31:58.602 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:31:58.602 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:31:58.637 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-04-22T08:31:58.249363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585473901,"num_objects":142950,"num_object_clones":0,"num_object_copies":285900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37187,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":142950,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150754,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7366,"num_bytes_recovered":26763881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2128460,"kb_used_data":1213296,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":280987060,"statfs":{"total":289910292480,"available":287730749440,"internally_reserved":0,"allocated":1242415104,"data_stored":1241716570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":937086062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,8,6,0,0,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16588800,"num_objects":-4050,"num_object_clones":0,"num_object_copies":-8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":-375,"num_objects_unfound":0,"num_objects_dirty":-4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":1536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002865"},"pg_stats":[{"pgid":"2.f","version":"37'11166","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.360764+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:31:55.360764+0000","last_peered":"2024-04-22T08:31:55.360764+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:31:55.360764+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36872192,"num_objects":9002,"num_object_clones":0,"num_object_copies":18004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11166,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5317},{"shards":"0,2","objects":2712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11209","reported_seq":11324,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.383045+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:31:55.383045+0000","last_peered":"2024-04-22T08:31:55.383045+0000","last_clean":"2024-04-22T08:31:55.383045+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:31:55.383045+0000","last_undegraded":"2024-04-22T08:31:55.383045+0000","last_fullsized":"2024-04-22T08:31:55.383045+0000","mapping_epoch":27,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36958208,"num_objects":9023,"num_object_clones":0,"num_object_copies":18046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8265,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'10951","reported_seq":11150,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.514859+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:31:55.514859+0000","last_peered":"2024-04-22T08:31:55.514859+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:31:55.514859+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503576,"num_objects":8913,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7778,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2738},{"shards":"2","objects":5195}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11155","reported_seq":11186,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.521717+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:31:55.521717+0000","last_peered":"2024-04-22T08:31:55.521717+0000","last_clean":"2024-04-22T08:31:55.521717+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:31:55.521717+0000","last_undegraded":"2024-04-22T08:31:55.521717+0000","last_fullsized":"2024-04-22T08:31:55.521717+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36696064,"num_objects":8959,"num_object_clones":0,"num_object_copies":17918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11155,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11029","reported_seq":11168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.371143+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:31:55.371143+0000","last_peered":"2024-04-22T08:31:55.371143+0000","last_clean":"2024-04-22T08:31:55.371143+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:31:55.371143+0000","last_undegraded":"2024-04-22T08:31:55.371143+0000","last_fullsized":"2024-04-22T08:31:55.371143+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11099","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.391869+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:31:55.391869+0000","last_peered":"2024-04-22T08:31:55.391869+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:31:55.391869+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36651008,"num_objects":8948,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5187},{"shards":"0,2","objects":2806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11002","reported_seq":18737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:54.015074+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:54.015074+0000","last_peered":"2024-04-22T08:31:54.015074+0000","last_clean":"2024-04-22T08:31:54.015074+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:54.015074+0000","last_undegraded":"2024-04-22T08:31:54.015074+0000","last_fullsized":"2024-04-22T08:31:54.015074+0000","mapping_epoch":36,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36298752,"num_objects":8862,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'10958","reported_seq":11103,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:55.302869+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:31:55.302869+0000","last_peered":"2024-04-22T08:31:55.302869+0000","last_clean":"2024-04-22T08:31:55.302869+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:31:55.302869+0000","last_undegraded":"2024-04-22T08:31:55.302869+0000","last_fullsized":"2024-04-22T08:31:55.302869+0000","mapping_epoch":27,"log_start":"28'900","ondisk_log_start":"28'900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":8848,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11253","reported_seq":11457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.556777+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:31:55.556777+0000","last_peered":"2024-04-22T08:31:55.556777+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:31:55.556777+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36734162,"num_objects":8970,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2778},{"shards":"2","objects":5122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11113","reported_seq":11409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.359764+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:31:55.359764+0000","last_peered":"2024-04-22T08:31:55.359764+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:31:55.359764+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37164242,"num_objects":9075,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5413},{"shards":"0,2","objects":2755}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11019","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.543469+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:31:55.543469+0000","last_peered":"2024-04-22T08:31:55.543469+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:31:55.543469+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":8910,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2698},{"shards":"2","objects":5221}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11027","reported_seq":16586,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:31:55.545488+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:31:55.545488+0000","last_peered":"2024-04-22T08:31:55.545488+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:31:55.545488+0000","last_undegraded":"2024-04-22T08:31:55.545488+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36484306,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11025,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2753,"num_bytes_recovered":11259904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1054},{"shards":"2","objects":7855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11107","reported_seq":11393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.439746+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:31:55.439746+0000","last_peered":"2024-04-22T08:31:55.439746+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:31:55.439746+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37007977,"num_objects":9036,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5418},{"shards":"0,2","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11006","reported_seq":11335,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.368416+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:31:55.368416+0000","last_peered":"2024-04-22T08:31:55.368416+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:31:55.368416+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":8744,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4984},{"shards":"0,2","objects":2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11096","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.302847+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:31:55.302847+0000","last_peered":"2024-04-22T08:31:55.302847+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:31:55.302847+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36627666,"num_objects":8944,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5310},{"shards":"0,2","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11089","reported_seq":11407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:31:55.317784+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:31:55.317784+0000","last_peered":"2024-04-22T08:31:55.317784+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:31:55.317784+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36510116,"num_objects":8917,"num_object_clones":0,"num_object_copies":17834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5123},{"shards":"0,2","objects":2799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":585473901,"num_objects":142950,"num_object_clones":0,"num_object_copies":285900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37187,"num_objects_misplaced":1054,"num_objects_unfound":0,"num_objects_dirty":142950,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150754,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7366,"num_bytes_recovered":26763881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241169920,"data_stored":1241088717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117055,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704268,"kb_used_data":407684,"kb_used_omap":4,"kb_used_meta":296571,"kb_avail":93667572,"statfs":{"total":96636764160,"available":95915593728,"internally_reserved":0,"allocated":417468416,"data_stored":417252934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":303689423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,3,0,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":5.4509999999999996}]}]},{"osd":1,"up_from":33,"seq":141733920817,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595860,"kb_used_data":315660,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93775980,"statfs":{"total":96636764160,"available":96026603520,"internally_reserved":0,"allocated":323235840,"data_stored":322957736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":286912204},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313283,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":828332,"kb_used_data":489952,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93543508,"statfs":{"total":96636764160,"available":95788552192,"internally_reserved":0,"allocated":501710848,"data_stored":501505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.83599999999999997,"15min":0.48999999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.3770000000000007,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":2.5630000000000002,"5min":1.214,"15min":0.98999999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":19.658000000000001,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.222},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":501366784,"data_stored":501363305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322912256,"data_stored":322842083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416890880,"data_stored":416883329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:01.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:02.546 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:02.547 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:02.601 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-22T08:32:02.250295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577118061,"num_objects":140910,"num_object_clones":0,"num_object_copies":281820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152794,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2137484,"kb_used_data":1204276,"kb_used_omap":12,"kb_used_meta":933171,"kb_avail":280978036,"statfs":{"total":289910292480,"available":287721508864,"internally_reserved":0,"allocated":1233178624,"data_stored":1232480090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":955567214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,5,15,7,0,6,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24391680,"num_objects":-5955,"num_object_clones":0,"num_object_copies":-11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-292,"num_objects_misplaced":-581,"num_objects_unfound":0,"num_objects_dirty":-5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":2379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002928"},"pg_stats":[{"pgid":"2.f","version":"37'11296","reported_seq":11604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.428602+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:00.428602+0000","last_peered":"2024-04-22T08:32:00.428602+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:00.428602+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36339712,"num_objects":8872,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11296,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5200},{"shards":"0,2","objects":2582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.552361+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:00.552361+0000","last_peered":"2024-04-22T08:32:00.552361+0000","last_clean":"2024-04-22T08:32:00.552361+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:00.552361+0000","last_undegraded":"2024-04-22T08:32:00.552361+0000","last_fullsized":"2024-04-22T08:32:00.552361+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":8884,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11106","reported_seq":11314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.528439+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:00.528439+0000","last_peered":"2024-04-22T08:32:00.528439+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:00.528439+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35868696,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7933,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2583},{"shards":"2","objects":5046}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11302","reported_seq":11333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.500748+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:00.500748+0000","last_peered":"2024-04-22T08:32:00.500748+0000","last_clean":"2024-04-22T08:32:00.500748+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:00.500748+0000","last_undegraded":"2024-04-22T08:32:00.500748+0000","last_fullsized":"2024-04-22T08:32:00.500748+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11161","reported_seq":11300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.357740+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:00.357740+0000","last_peered":"2024-04-22T08:32:00.357740+0000","last_clean":"2024-04-22T08:32:00.357740+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:00.357740+0000","last_undegraded":"2024-04-22T08:32:00.357740+0000","last_fullsized":"2024-04-22T08:32:00.357740+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11214","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.275403+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:00.275403+0000","last_peered":"2024-04-22T08:32:00.275403+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:00.275403+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179968,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8083,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11114","reported_seq":18850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:58.937434+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:58.937434+0000","last_peered":"2024-04-22T08:31:58.937434+0000","last_clean":"2024-04-22T08:31:58.937434+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:58.937434+0000","last_undegraded":"2024-04-22T08:31:58.937434+0000","last_fullsized":"2024-04-22T08:31:58.937434+0000","mapping_epoch":36,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11079","reported_seq":11224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.325211+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:00.325211+0000","last_peered":"2024-04-22T08:32:00.325211+0000","last_clean":"2024-04-22T08:32:00.325211+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:00.325211+0000","last_undegraded":"2024-04-22T08:32:00.325211+0000","last_fullsized":"2024-04-22T08:32:00.325211+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35745792,"num_objects":8727,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11367","reported_seq":11575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.527513+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:00.527513+0000","last_peered":"2024-04-22T08:32:00.527513+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:00.527513+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36267218,"num_objects":8856,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2664},{"shards":"2","objects":5010}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11249","reported_seq":11562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.409339+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:00.409339+0000","last_peered":"2024-04-22T08:32:00.409339+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:00.409339+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607186,"num_objects":8939,"num_object_clones":0,"num_object_copies":17878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":2619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11159","reported_seq":11442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.383126+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:00.383126+0000","last_peered":"2024-04-22T08:32:00.383126+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:00.383126+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2558},{"shards":"2","objects":5088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11140","reported_seq":17111,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:00.573927+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:00.573927+0000","last_peered":"2024-04-22T08:32:00.573927+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:00.573927+0000","last_undegraded":"2024-04-22T08:32:00.573927+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36021458,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":12103680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":848},{"shards":"2","objects":7948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11227","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.443653+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:00.443653+0000","last_peered":"2024-04-22T08:32:00.443653+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:00.443653+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36516457,"num_objects":8916,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11226,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5316},{"shards":"0,2","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11131","reported_seq":11480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.422026+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:00.422026+0000","last_peered":"2024-04-22T08:32:00.422026+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:00.422026+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4869},{"shards":"0,2","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11213","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410206+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:00.410206+0000","last_peered":"2024-04-22T08:32:00.410206+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:00.410206+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36148434,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5206},{"shards":"0,2","objects":2513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11213","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410578+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:00.410578+0000","last_peered":"2024-04-22T08:32:00.410578+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:00.410578+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36002212,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5010},{"shards":"0,2","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":577118061,"num_objects":140910,"num_object_clones":0,"num_object_copies":281820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152794,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230532608,"data_stored":1230451405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117056,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705900,"kb_used_data":403556,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93665940,"statfs":{"total":96636764160,"available":95913922560,"internally_reserved":0,"allocated":413241344,"data_stored":413025862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,3,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920818,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604016,"kb_used_data":315624,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93767824,"statfs":{"total":96636764160,"available":96018251776,"internally_reserved":0,"allocated":323198976,"data_stored":322920872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.498},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.435},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827568,"kb_used_data":485096,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93544272,"statfs":{"total":96636764160,"available":95789334528,"internally_reserved":0,"allocated":496738304,"data_stored":496533356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,9,1,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.312},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.378}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495112192,"data_stored":495108713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322891776,"data_stored":322821603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412528640,"data_stored":412521089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:02.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:03.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:03.271 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:03.310 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-22T08:32:02.250295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577118061,"num_objects":140910,"num_object_clones":0,"num_object_copies":281820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152794,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2137484,"kb_used_data":1204276,"kb_used_omap":12,"kb_used_meta":933171,"kb_avail":280978036,"statfs":{"total":289910292480,"available":287721508864,"internally_reserved":0,"allocated":1233178624,"data_stored":1232480090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":955567214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,5,15,7,0,6,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24391680,"num_objects":-5955,"num_object_clones":0,"num_object_copies":-11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-292,"num_objects_misplaced":-581,"num_objects_unfound":0,"num_objects_dirty":-5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":2379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002928"},"pg_stats":[{"pgid":"2.f","version":"37'11296","reported_seq":11604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.428602+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:00.428602+0000","last_peered":"2024-04-22T08:32:00.428602+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:00.428602+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36339712,"num_objects":8872,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11296,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5200},{"shards":"0,2","objects":2582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.552361+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:00.552361+0000","last_peered":"2024-04-22T08:32:00.552361+0000","last_clean":"2024-04-22T08:32:00.552361+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:00.552361+0000","last_undegraded":"2024-04-22T08:32:00.552361+0000","last_fullsized":"2024-04-22T08:32:00.552361+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":8884,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11106","reported_seq":11314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.528439+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:00.528439+0000","last_peered":"2024-04-22T08:32:00.528439+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:00.528439+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35868696,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7933,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2583},{"shards":"2","objects":5046}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11302","reported_seq":11333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.500748+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:00.500748+0000","last_peered":"2024-04-22T08:32:00.500748+0000","last_clean":"2024-04-22T08:32:00.500748+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:00.500748+0000","last_undegraded":"2024-04-22T08:32:00.500748+0000","last_fullsized":"2024-04-22T08:32:00.500748+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11161","reported_seq":11300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.357740+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:00.357740+0000","last_peered":"2024-04-22T08:32:00.357740+0000","last_clean":"2024-04-22T08:32:00.357740+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:00.357740+0000","last_undegraded":"2024-04-22T08:32:00.357740+0000","last_fullsized":"2024-04-22T08:32:00.357740+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11214","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.275403+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:00.275403+0000","last_peered":"2024-04-22T08:32:00.275403+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:00.275403+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179968,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8083,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11114","reported_seq":18850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:31:58.937434+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:31:58.937434+0000","last_peered":"2024-04-22T08:31:58.937434+0000","last_clean":"2024-04-22T08:31:58.937434+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:31:58.937434+0000","last_undegraded":"2024-04-22T08:31:58.937434+0000","last_fullsized":"2024-04-22T08:31:58.937434+0000","mapping_epoch":36,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11079","reported_seq":11224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.325211+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:00.325211+0000","last_peered":"2024-04-22T08:32:00.325211+0000","last_clean":"2024-04-22T08:32:00.325211+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:00.325211+0000","last_undegraded":"2024-04-22T08:32:00.325211+0000","last_fullsized":"2024-04-22T08:32:00.325211+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35745792,"num_objects":8727,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11367","reported_seq":11575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.527513+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:00.527513+0000","last_peered":"2024-04-22T08:32:00.527513+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:00.527513+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36267218,"num_objects":8856,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2664},{"shards":"2","objects":5010}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11249","reported_seq":11562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.409339+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:00.409339+0000","last_peered":"2024-04-22T08:32:00.409339+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:00.409339+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607186,"num_objects":8939,"num_object_clones":0,"num_object_copies":17878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":2619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11159","reported_seq":11442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.383126+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:00.383126+0000","last_peered":"2024-04-22T08:32:00.383126+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:00.383126+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2558},{"shards":"2","objects":5088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.147054+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:09.147054+0000","last_peered":"2024-04-22T08:30:09.147054+0000","last_clean":"2024-04-22T08:30:09.147054+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:09.147054+0000","last_undegraded":"2024-04-22T08:30:09.147054+0000","last_fullsized":"2024-04-22T08:30:09.147054+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11140","reported_seq":17111,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:00.573927+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:00.573927+0000","last_peered":"2024-04-22T08:32:00.573927+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:00.573927+0000","last_undegraded":"2024-04-22T08:32:00.573927+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36021458,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":12103680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":848},{"shards":"2","objects":7948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11227","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.443653+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:00.443653+0000","last_peered":"2024-04-22T08:32:00.443653+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:00.443653+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36516457,"num_objects":8916,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11226,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5316},{"shards":"0,2","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.150926+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:09.150926+0000","last_peered":"2024-04-22T08:30:09.150926+0000","last_clean":"2024-04-22T08:30:09.150926+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:09.150926+0000","last_undegraded":"2024-04-22T08:30:09.150926+0000","last_fullsized":"2024-04-22T08:30:09.150926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11131","reported_seq":11480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.422026+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:00.422026+0000","last_peered":"2024-04-22T08:32:00.422026+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:00.422026+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4869},{"shards":"0,2","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11213","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410206+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:00.410206+0000","last_peered":"2024-04-22T08:32:00.410206+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:00.410206+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36148434,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5206},{"shards":"0,2","objects":2513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143965+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:09.143965+0000","last_peered":"2024-04-22T08:30:09.143965+0000","last_clean":"2024-04-22T08:30:09.143965+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:09.143965+0000","last_undegraded":"2024-04-22T08:30:09.143965+0000","last_fullsized":"2024-04-22T08:30:09.143965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11213","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410578+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:00.410578+0000","last_peered":"2024-04-22T08:32:00.410578+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:00.410578+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36002212,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5010},{"shards":"0,2","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":577118061,"num_objects":140910,"num_object_clones":0,"num_object_copies":281820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152794,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230532608,"data_stored":1230451405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117056,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705900,"kb_used_data":403556,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93665940,"statfs":{"total":96636764160,"available":95913922560,"internally_reserved":0,"allocated":413241344,"data_stored":413025862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,3,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920818,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604016,"kb_used_data":315624,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93767824,"statfs":{"total":96636764160,"available":96018251776,"internally_reserved":0,"allocated":323198976,"data_stored":322920872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":295300812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.71999999999999997,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6559999999999997,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.498},{"interface":"front","average":{"1min":0.753,"5min":0.90200000000000002,"15min":0.94899999999999995},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.1079999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":1.7010000000000001,"15min":1.4830000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.989000000000001,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.435},{"interface":"front","average":{"1min":2.5419999999999998,"5min":1.466,"15min":1.256},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":20.149000000000001,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827568,"kb_used_data":485096,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93544272,"statfs":{"total":96636764160,"available":95789334528,"internally_reserved":0,"allocated":496738304,"data_stored":496533356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,9,1,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.312},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.378}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495112192,"data_stored":495108713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322891776,"data_stored":322821603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412528640,"data_stored":412521089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:03.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:03.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'11296", 'reported_seq': 11604, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.428602+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:00.428602+0000', 'last_peered': '2024-04-22T08:32:00.428602+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:00.428602+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36339712, 'num_objects': 8872, 'num_object_clones': 0, 'num_object_copies': 17744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3672, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11296, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5200}, {'shards': '0,2', 'objects': 2582}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:03.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'11106", 'reported_seq': 11314, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.528439+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:00.528439+0000', 'last_peered': '2024-04-22T08:32:00.528439+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:00.528439+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35868696, 'num_objects': 8758, 'num_object_clones': 0, 'num_object_copies': 17516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3713, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8758, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7933, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2583}, {'shards': '2', 'objects': 5046}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'11214", 'reported_seq': 11633, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.275403+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:32:00.275403+0000', 'last_peered': '2024-04-22T08:32:00.275403+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:00.275403+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36179968, 'num_objects': 8833, 'num_object_clones': 0, 'num_object_copies': 17666, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3753, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8083, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 129, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5082}, {'shards': '0,2', 'objects': 2691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'11367", 'reported_seq': 11575, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.527513+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:00.527513+0000', 'last_peered': '2024-04-22T08:32:00.527513+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:00.527513+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36267218, 'num_objects': 8856, 'num_object_clones': 0, 'num_object_copies': 17712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3848, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11307, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2664}, {'shards': '2', 'objects': 5010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'11249", 'reported_seq': 11562, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.409339+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:00.409339+0000', 'last_peered': '2024-04-22T08:32:00.409339+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:00.409339+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36607186, 'num_objects': 8939, 'num_object_clones': 0, 'num_object_copies': 17878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3655, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11247, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 121, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5286}, {'shards': '0,2', 'objects': 2619}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'11159", 'reported_seq': 11442, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.383126+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:00.383126+0000', 'last_peered': '2024-04-22T08:32:00.383126+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:00.383126+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35921920, 'num_objects': 8770, 'num_object_clones': 0, 'num_object_copies': 17540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3684, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7613, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2558}, {'shards': '2', 'objects': 5088}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:32:03.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'11140", 'reported_seq': 17111, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:32:00.573927+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:32:00.573927+0000', 'last_peered': '2024-04-22T08:32:00.573927+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:32:00.573927+0000', 'last_undegraded': '2024-04-22T08:32:00.573927+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36021458, 'num_objects': 8796, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 848, 'num_objects_unfound': 0, 'num_objects_dirty': 8796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11138, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2959, 'num_bytes_recovered': 12103680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 848}, {'shards': '2', 'objects': 7948}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'11227", 'reported_seq': 11549, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.443653+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:00.443653+0000', 'last_peered': '2024-04-22T08:32:00.443653+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:00.443653+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36516457, 'num_objects': 8916, 'num_object_clones': 0, 'num_object_copies': 17832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11226, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 123, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5316}, {'shards': '0,2', 'objects': 2566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'11131", 'reported_seq': 11480, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.422026+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:00.422026+0000', 'last_peered': '2024-04-22T08:32:00.422026+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:00.422026+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35303424, 'num_objects': 8619, 'num_object_clones': 0, 'num_object_copies': 17238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3753, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7362, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4869}, {'shards': '0,2', 'objects': 2585}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'11213", 'reported_seq': 11458, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.410206+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:00.410206+0000', 'last_peered': '2024-04-22T08:32:00.410206+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:00.410206+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36148434, 'num_objects': 8827, 'num_object_clones': 0, 'num_object_copies': 17654, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3623, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11211, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5206}, {'shards': '0,2', 'objects': 2513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:03.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'11213", 'reported_seq': 11549, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:00.410578+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:00.410578+0000', 'last_peered': '2024-04-22T08:32:00.410578+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:00.410578+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36002212, 'num_objects': 8793, 'num_object_clones': 0, 'num_object_copies': 17586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3786, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11210, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5010}, {'shards': '0,2', 'objects': 2675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:03.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:03.836 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:03.838 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37088/281820 objects degraded (13.160%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":358,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":140910,"data_bytes":577118061,"bytes_used":2188783616,"bytes_avail":287721508864,"bytes_total":289910292480,"degraded_objects":37088,"degraded_total":281820,"degraded_ratio":0.13160173160173161,"misplaced_objects":848,"misplaced_total":281820,"misplaced_ratio":0.0030090128450784188,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":198266,"recovering_keys_per_sec":0,"num_objects_recovered":873,"num_bytes_recovered":2379776,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":496},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:03.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:03.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:04.555 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:04.555 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:04.625 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-22T08:32:04.250874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576581485,"num_objects":140779,"num_object_clones":0,"num_object_copies":281558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152925,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160650,"ondisk_log_size":160650,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2159596,"kb_used_data":1204244,"kb_used_omap":12,"kb_used_meta":955315,"kb_avail":280955924,"statfs":{"total":289910292480,"available":287698866176,"internally_reserved":0,"allocated":1233145856,"data_stored":1232447322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":978242670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,4,6,12,3,0,6,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17133568,"num_objects":-4183,"num_object_clones":0,"num_object_copies":-8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":-414,"num_objects_unfound":0,"num_objects_dirty":-4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003003"},"pg_stats":[{"pgid":"2.f","version":"37'11296","reported_seq":11604,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.428602+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:00.428602+0000","last_peered":"2024-04-22T08:32:00.428602+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:00.428602+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36339712,"num_objects":8872,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11296,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5200},{"shards":"0,2","objects":2582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.552361+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:00.552361+0000","last_peered":"2024-04-22T08:32:00.552361+0000","last_clean":"2024-04-22T08:32:00.552361+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:00.552361+0000","last_undegraded":"2024-04-22T08:32:00.552361+0000","last_fullsized":"2024-04-22T08:32:00.552361+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":8884,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11106","reported_seq":11314,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.528439+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:00.528439+0000","last_peered":"2024-04-22T08:32:00.528439+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:00.528439+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35868696,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7933,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2583},{"shards":"2","objects":5046}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11302","reported_seq":11333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.500748+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:00.500748+0000","last_peered":"2024-04-22T08:32:00.500748+0000","last_clean":"2024-04-22T08:32:00.500748+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:00.500748+0000","last_undegraded":"2024-04-22T08:32:00.500748+0000","last_fullsized":"2024-04-22T08:32:00.500748+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11161","reported_seq":11300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.357740+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:00.357740+0000","last_peered":"2024-04-22T08:32:00.357740+0000","last_clean":"2024-04-22T08:32:00.357740+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:00.357740+0000","last_undegraded":"2024-04-22T08:32:00.357740+0000","last_fullsized":"2024-04-22T08:32:00.357740+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":8758,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11214","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.275403+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:00.275403+0000","last_peered":"2024-04-22T08:32:00.275403+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:00.275403+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36179968,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8083,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11245","reported_seq":18981,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:04.165188+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:04.165188+0000","last_peered":"2024-04-22T08:32:04.165188+0000","last_clean":"2024-04-22T08:32:04.165188+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:04.165188+0000","last_undegraded":"2024-04-22T08:32:04.165188+0000","last_fullsized":"2024-04-22T08:32:04.165188+0000","mapping_epoch":36,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11079","reported_seq":11224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:00.325211+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:00.325211+0000","last_peered":"2024-04-22T08:32:00.325211+0000","last_clean":"2024-04-22T08:32:00.325211+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:00.325211+0000","last_undegraded":"2024-04-22T08:32:00.325211+0000","last_fullsized":"2024-04-22T08:32:00.325211+0000","mapping_epoch":27,"log_start":"28'1000","ondisk_log_start":"28'1000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35745792,"num_objects":8727,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11367","reported_seq":11575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.527513+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:00.527513+0000","last_peered":"2024-04-22T08:32:00.527513+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:00.527513+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36267218,"num_objects":8856,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2664},{"shards":"2","objects":5010}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11249","reported_seq":11562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.409339+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:00.409339+0000","last_peered":"2024-04-22T08:32:00.409339+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:00.409339+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607186,"num_objects":8939,"num_object_clones":0,"num_object_copies":17878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":2619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11159","reported_seq":11442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.383126+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:00.383126+0000","last_peered":"2024-04-22T08:32:00.383126+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:00.383126+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2558},{"shards":"2","objects":5088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11140","reported_seq":17111,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:00.573927+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:00.573927+0000","last_peered":"2024-04-22T08:32:00.573927+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:00.573927+0000","last_undegraded":"2024-04-22T08:32:00.573927+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36021458,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":12103680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":848},{"shards":"2","objects":7948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11227","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.443653+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:00.443653+0000","last_peered":"2024-04-22T08:32:00.443653+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:00.443653+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36516457,"num_objects":8916,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11226,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5316},{"shards":"0,2","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11131","reported_seq":11480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.422026+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:00.422026+0000","last_peered":"2024-04-22T08:32:00.422026+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:00.422026+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4869},{"shards":"0,2","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11213","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410206+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:00.410206+0000","last_peered":"2024-04-22T08:32:00.410206+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:00.410206+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36148434,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5206},{"shards":"0,2","objects":2513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11213","reported_seq":11549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:00.410578+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:00.410578+0000","last_peered":"2024-04-22T08:32:00.410578+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:00.410578+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36002212,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5010},{"shards":"0,2","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":576581485,"num_objects":140779,"num_object_clones":0,"num_object_copies":281558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37088,"num_objects_misplaced":848,"num_objects_unfound":0,"num_objects_dirty":140779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152925,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7671,"num_bytes_recovered":27607657,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230508032,"data_stored":1230426829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160650,"ondisk_log_size":160650,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117056,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705900,"kb_used_data":403556,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93665940,"statfs":{"total":96636764160,"available":95913922560,"internally_reserved":0,"allocated":413241344,"data_stored":413025862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,3,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.04,"15min":0.96999999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.72499999999999998,"15min":0.625},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4029999999999996,"5min":5.7309999999999999,"15min":5.7309999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920819,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626128,"kb_used_data":315592,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93745712,"statfs":{"total":96636764160,"available":95995609088,"internally_reserved":0,"allocated":323166208,"data_stored":322888104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":317976268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.245}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827568,"kb_used_data":485096,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93544272,"statfs":{"total":96636764160,"available":95789334528,"internally_reserved":0,"allocated":496738304,"data_stored":496533356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":350678739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,9,1,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.312},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.378}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495112192,"data_stored":495108713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322867200,"data_stored":322797027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412528640,"data_stored":412521089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:07.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:08.344 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:08.346 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:08.402 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-22T08:32:08.259359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568971117,"num_objects":138921,"num_object_clones":0,"num_object_copies":277842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2157840,"kb_used_data":1194296,"kb_used_omap":12,"kb_used_meta":963507,"kb_avail":280957680,"statfs":{"total":289910292480,"available":287700664320,"internally_reserved":0,"allocated":1222959104,"data_stored":1222260570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":986631278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,10,2,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16502784,"num_objects":-4029,"num_object_clones":0,"num_object_copies":-8058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-447,"num_objects_unfound":0,"num_objects_dirty":-4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":1830912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010505"},"pg_stats":[{"pgid":"2.f","version":"37'11419","reported_seq":11739,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.479541+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:05.479541+0000","last_peered":"2024-04-22T08:32:05.479541+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:05.479541+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11419,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5083},{"shards":"0,2","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11482","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.584028+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:05.584028+0000","last_peered":"2024-04-22T08:32:05.584028+0000","last_clean":"2024-04-22T08:32:05.584028+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:05.584028+0000","last_undegraded":"2024-04-22T08:32:05.584028+0000","last_fullsized":"2024-04-22T08:32:05.584028+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11264","reported_seq":11501,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.506801+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:05.506801+0000","last_peered":"2024-04-22T08:32:05.506801+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:05.506801+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35221528,"num_objects":8600,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8091,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2425},{"shards":"2","objects":4901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11431","reported_seq":11462,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.575444+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:05.575444+0000","last_peered":"2024-04-22T08:32:05.575444+0000","last_clean":"2024-04-22T08:32:05.575444+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:05.575444+0000","last_undegraded":"2024-04-22T08:32:05.575444+0000","last_fullsized":"2024-04-22T08:32:05.575444+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35565568,"num_objects":8683,"num_object_clones":0,"num_object_copies":17366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11300","reported_seq":11439,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.383324+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:05.383324+0000","last_peered":"2024-04-22T08:32:05.383324+0000","last_clean":"2024-04-22T08:32:05.383324+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:05.383324+0000","last_undegraded":"2024-04-22T08:32:05.383324+0000","last_fullsized":"2024-04-22T08:32:05.383324+0000","mapping_epoch":27,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11332","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.415405+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:05.415405+0000","last_peered":"2024-04-22T08:32:05.415405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:05.415405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8201,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4974},{"shards":"0,2","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11245","reported_seq":18981,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:04.165188+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:04.165188+0000","last_peered":"2024-04-22T08:32:04.165188+0000","last_clean":"2024-04-22T08:32:04.165188+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:04.165188+0000","last_undegraded":"2024-04-22T08:32:04.165188+0000","last_fullsized":"2024-04-22T08:32:04.165188+0000","mapping_epoch":36,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11198","reported_seq":11343,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.361086+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:05.361086+0000","last_peered":"2024-04-22T08:32:05.361086+0000","last_clean":"2024-04-22T08:32:05.361086+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:05.361086+0000","last_undegraded":"2024-04-22T08:32:05.361086+0000","last_fullsized":"2024-04-22T08:32:05.361086+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":8608,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11494","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.596384+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:05.596384+0000","last_peered":"2024-04-22T08:32:05.596384+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:05.596384+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35747026,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2537},{"shards":"2","objects":4890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11383","reported_seq":11717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.368125+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:05.368125+0000","last_peered":"2024-04-22T08:32:05.368125+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:05.368125+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058322,"num_objects":8805,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5162},{"shards":"0,2","objects":2485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11266","reported_seq":11567,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.549620+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:05.549620+0000","last_peered":"2024-04-22T08:32:05.549620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:05.549620+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2451},{"shards":"2","objects":4990}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11264","reported_seq":17716,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:05.569979+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:05.569979+0000","last_peered":"2024-04-22T08:32:05.569979+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:05.569979+0000","last_undegraded":"2024-04-22T08:32:05.569979+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513554,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3200,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":8065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11340","reported_seq":11682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.394315+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:05.394315+0000","last_peered":"2024-04-22T08:32:05.394315+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:05.394315+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36053609,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11240","reported_seq":11601,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.412138+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:05.412138+0000","last_peered":"2024-04-22T08:32:05.412138+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:05.412138+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":8510,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7471,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4766},{"shards":"0,2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11316","reported_seq":11565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.438173+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:05.438173+0000","last_peered":"2024-04-22T08:32:05.438173+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:05.438173+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35726546,"num_objects":8724,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5105},{"shards":"0,2","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11334","reported_seq":11696,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.474345+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:05.474345+0000","last_peered":"2024-04-22T08:32:05.474345+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:05.474345+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35506596,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":568971117,"num_objects":138921,"num_object_clones":0,"num_object_copies":277842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220280320,"data_stored":1220199117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117057,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705984,"kb_used_data":399544,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93665856,"statfs":{"total":96636764160,"available":95913836544,"internally_reserved":0,"allocated":409133056,"data_stored":408917574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.9670000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920819,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626128,"kb_used_data":315592,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93745712,"statfs":{"total":96636764160,"available":95995609088,"internally_reserved":0,"allocated":323166208,"data_stored":322888104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":317976268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.245}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313285,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825728,"kb_used_data":479160,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93546112,"statfs":{"total":96636764160,"available":95791218688,"internally_reserved":0,"allocated":490659840,"data_stored":490454892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,1,0,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":3.585},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":3.6230000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":489123840,"data_stored":489120361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322867200,"data_stored":322797027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408289280,"data_stored":408281729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:08.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:09.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:09.050 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:09.090 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-22T08:32:08.259359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568971117,"num_objects":138921,"num_object_clones":0,"num_object_copies":277842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2157840,"kb_used_data":1194296,"kb_used_omap":12,"kb_used_meta":963507,"kb_avail":280957680,"statfs":{"total":289910292480,"available":287700664320,"internally_reserved":0,"allocated":1222959104,"data_stored":1222260570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":986631278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,10,2,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16502784,"num_objects":-4029,"num_object_clones":0,"num_object_copies":-8058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-447,"num_objects_unfound":0,"num_objects_dirty":-4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":1830912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010505"},"pg_stats":[{"pgid":"2.f","version":"37'11419","reported_seq":11739,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.479541+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:05.479541+0000","last_peered":"2024-04-22T08:32:05.479541+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:05.479541+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11419,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5083},{"shards":"0,2","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11482","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.584028+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:05.584028+0000","last_peered":"2024-04-22T08:32:05.584028+0000","last_clean":"2024-04-22T08:32:05.584028+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:05.584028+0000","last_undegraded":"2024-04-22T08:32:05.584028+0000","last_fullsized":"2024-04-22T08:32:05.584028+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11264","reported_seq":11501,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.506801+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:05.506801+0000","last_peered":"2024-04-22T08:32:05.506801+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:05.506801+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35221528,"num_objects":8600,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8091,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2425},{"shards":"2","objects":4901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11431","reported_seq":11462,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.575444+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:05.575444+0000","last_peered":"2024-04-22T08:32:05.575444+0000","last_clean":"2024-04-22T08:32:05.575444+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:05.575444+0000","last_undegraded":"2024-04-22T08:32:05.575444+0000","last_fullsized":"2024-04-22T08:32:05.575444+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35565568,"num_objects":8683,"num_object_clones":0,"num_object_copies":17366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11300","reported_seq":11439,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.383324+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:05.383324+0000","last_peered":"2024-04-22T08:32:05.383324+0000","last_clean":"2024-04-22T08:32:05.383324+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:05.383324+0000","last_undegraded":"2024-04-22T08:32:05.383324+0000","last_fullsized":"2024-04-22T08:32:05.383324+0000","mapping_epoch":27,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11332","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.415405+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:05.415405+0000","last_peered":"2024-04-22T08:32:05.415405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:05.415405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8201,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4974},{"shards":"0,2","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11245","reported_seq":18981,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:04.165188+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:04.165188+0000","last_peered":"2024-04-22T08:32:04.165188+0000","last_clean":"2024-04-22T08:32:04.165188+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:04.165188+0000","last_undegraded":"2024-04-22T08:32:04.165188+0000","last_fullsized":"2024-04-22T08:32:04.165188+0000","mapping_epoch":36,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11198","reported_seq":11343,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.361086+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:05.361086+0000","last_peered":"2024-04-22T08:32:05.361086+0000","last_clean":"2024-04-22T08:32:05.361086+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:05.361086+0000","last_undegraded":"2024-04-22T08:32:05.361086+0000","last_fullsized":"2024-04-22T08:32:05.361086+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":8608,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11494","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.596384+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:05.596384+0000","last_peered":"2024-04-22T08:32:05.596384+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:05.596384+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35747026,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2537},{"shards":"2","objects":4890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11383","reported_seq":11717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.368125+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:05.368125+0000","last_peered":"2024-04-22T08:32:05.368125+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:05.368125+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058322,"num_objects":8805,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5162},{"shards":"0,2","objects":2485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11266","reported_seq":11567,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.549620+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:05.549620+0000","last_peered":"2024-04-22T08:32:05.549620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:05.549620+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2451},{"shards":"2","objects":4990}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11264","reported_seq":17716,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:05.569979+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:05.569979+0000","last_peered":"2024-04-22T08:32:05.569979+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:05.569979+0000","last_undegraded":"2024-04-22T08:32:05.569979+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513554,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3200,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":8065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11340","reported_seq":11682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.394315+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:05.394315+0000","last_peered":"2024-04-22T08:32:05.394315+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:05.394315+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36053609,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11240","reported_seq":11601,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.412138+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:05.412138+0000","last_peered":"2024-04-22T08:32:05.412138+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:05.412138+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":8510,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7471,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4766},{"shards":"0,2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11316","reported_seq":11565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.438173+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:05.438173+0000","last_peered":"2024-04-22T08:32:05.438173+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:05.438173+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35726546,"num_objects":8724,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5105},{"shards":"0,2","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11334","reported_seq":11696,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.474345+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:05.474345+0000","last_peered":"2024-04-22T08:32:05.474345+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:05.474345+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35506596,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":568971117,"num_objects":138921,"num_object_clones":0,"num_object_copies":277842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220280320,"data_stored":1220199117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160908,"ondisk_log_size":160908,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117057,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705984,"kb_used_data":399544,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93665856,"statfs":{"total":96636764160,"available":95913836544,"internally_reserved":0,"allocated":409133056,"data_stored":408917574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.9670000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920819,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626128,"kb_used_data":315592,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93745712,"statfs":{"total":96636764160,"available":95995609088,"internally_reserved":0,"allocated":323166208,"data_stored":322888104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":317976268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.245}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313285,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825728,"kb_used_data":479160,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93546112,"statfs":{"total":96636764160,"available":95791218688,"internally_reserved":0,"allocated":490659840,"data_stored":490454892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,1,0,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":3.585},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":3.6230000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":489123840,"data_stored":489120361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322867200,"data_stored":322797027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408289280,"data_stored":408281729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:09.093 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:09.093 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'11419", 'reported_seq': 11739, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.479541+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:05.479541+0000', 'last_peered': '2024-04-22T08:32:05.479541+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:05.479541+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35835904, 'num_objects': 8749, 'num_object_clones': 0, 'num_object_copies': 17498, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3666, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11419, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 128, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5083}, {'shards': '0,2', 'objects': 2459}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.093 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:09.093 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'11264", 'reported_seq': 11501, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.506801+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:05.506801+0000', 'last_peered': '2024-04-22T08:32:05.506801+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:05.506801+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35221528, 'num_objects': 8600, 'num_object_clones': 0, 'num_object_copies': 17200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3700, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8600, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8091, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2425}, {'shards': '2', 'objects': 4901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:09.094 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'11332", 'reported_seq': 11771, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.415405+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:32:05.415405+0000', 'last_peered': '2024-04-22T08:32:05.415405+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:05.415405+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35696640, 'num_objects': 8715, 'num_object_clones': 0, 'num_object_copies': 17430, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3743, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8201, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 139, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4974}, {'shards': '0,2', 'objects': 2573}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'11494", 'reported_seq': 11716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.596384+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:05.596384+0000', 'last_peered': '2024-04-22T08:32:05.596384+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:05.596384+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35747026, 'num_objects': 8729, 'num_object_clones': 0, 'num_object_copies': 17458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3841, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11434, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2537}, {'shards': '2', 'objects': 4890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'11383", 'reported_seq': 11717, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.368125+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:05.368125+0000', 'last_peered': '2024-04-22T08:32:05.368125+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:05.368125+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36058322, 'num_objects': 8805, 'num_object_clones': 0, 'num_object_copies': 17610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3645, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11381, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 131, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5162}, {'shards': '0,2', 'objects': 2485}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.095 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:09.096 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'11266", 'reported_seq': 11567, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.549620+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:05.549620+0000', 'last_peered': '2024-04-22T08:32:05.549620+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:05.549620+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35483648, 'num_objects': 8663, 'num_object_clones': 0, 'num_object_copies': 17326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3675, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7720, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2451}, {'shards': '2', 'objects': 4990}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:09.096 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:32:09.096 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'11264", 'reported_seq': 17716, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:32:05.569979+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:32:05.569979+0000', 'last_peered': '2024-04-22T08:32:05.569979+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:32:05.569979+0000', 'last_undegraded': '2024-04-22T08:32:05.569979+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35513554, 'num_objects': 8672, 'num_object_clones': 0, 'num_object_copies': 17344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 607, 'num_objects_unfound': 0, 'num_objects_dirty': 8672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11262, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3200, 'num_bytes_recovered': 13090816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 607}, {'shards': '2', 'objects': 8065}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:09.097 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:09.097 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'11340", 'reported_seq': 11682, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.394315+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:05.394315+0000', 'last_peered': '2024-04-22T08:32:05.394315+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:05.394315+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36053609, 'num_objects': 8803, 'num_object_clones': 0, 'num_object_copies': 17606, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3591, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11339, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 133, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5213}, {'shards': '0,2', 'objects': 2453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.097 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:09.097 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'11240", 'reported_seq': 11601, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.412138+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:05.412138+0000', 'last_peered': '2024-04-22T08:32:05.412138+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:05.412138+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34856960, 'num_objects': 8510, 'num_object_clones': 0, 'num_object_copies': 17020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3747, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7471, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 93, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4766}, {'shards': '0,2', 'objects': 2476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.098 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:09.098 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'11316", 'reported_seq': 11565, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.438173+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:05.438173+0000', 'last_peered': '2024-04-22T08:32:05.438173+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:05.438173+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35726546, 'num_objects': 8724, 'num_object_clones': 0, 'num_object_copies': 17448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3621, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11314, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 88, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5105}, {'shards': '0,2', 'objects': 2410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.098 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:09.098 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'11334", 'reported_seq': 11696, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:05.474345+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:05.474345+0000', 'last_peered': '2024-04-22T08:32:05.474345+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:05.474345+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35506596, 'num_objects': 8672, 'num_object_clones': 0, 'num_object_copies': 17344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3775, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11331, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 114, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4900}, {'shards': '0,2', 'objects': 2554}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:09.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:09.967 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37004/277842 objects degraded (13.318%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":138921,"data_bytes":568971117,"bytes_used":2209628160,"bytes_avail":287700664320,"bytes_total":289910292480,"degraded_objects":37004,"degraded_total":277842,"degraded_ratio":0.13318360794984199,"misplaced_objects":607,"misplaced_total":277842,"misplaced_ratio":0.0021846948985394577,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":152442,"recovering_keys_per_sec":0,"num_objects_recovered":630,"num_bytes_recovered":1830912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":335},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:10.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:10.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:10.896 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:10.897 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:10.935 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-04-22T08:32:10.259854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568500077,"num_objects":138806,"num_object_clones":0,"num_object_copies":277612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154898,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2149756,"kb_used_data":1194404,"kb_used_omap":12,"kb_used_meta":955315,"kb_avail":280965764,"statfs":{"total":289910292480,"available":287708942336,"internally_reserved":0,"allocated":1223069696,"data_stored":1222371162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":978242670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,6,2,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16973824,"num_objects":-4144,"num_object_clones":0,"num_object_copies":-8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-447,"num_objects_unfound":0,"num_objects_dirty":-4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4144,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":1830912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010490"},"pg_stats":[{"pgid":"2.f","version":"37'11419","reported_seq":11739,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.479541+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:05.479541+0000","last_peered":"2024-04-22T08:32:05.479541+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:05.479541+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11419,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5083},{"shards":"0,2","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11482","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.584028+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:05.584028+0000","last_peered":"2024-04-22T08:32:05.584028+0000","last_clean":"2024-04-22T08:32:05.584028+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:05.584028+0000","last_undegraded":"2024-04-22T08:32:05.584028+0000","last_fullsized":"2024-04-22T08:32:05.584028+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11264","reported_seq":11501,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.506801+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:05.506801+0000","last_peered":"2024-04-22T08:32:05.506801+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:05.506801+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35221528,"num_objects":8600,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8091,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2425},{"shards":"2","objects":4901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11431","reported_seq":11462,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.575444+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:05.575444+0000","last_peered":"2024-04-22T08:32:05.575444+0000","last_clean":"2024-04-22T08:32:05.575444+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:05.575444+0000","last_undegraded":"2024-04-22T08:32:05.575444+0000","last_fullsized":"2024-04-22T08:32:05.575444+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35565568,"num_objects":8683,"num_object_clones":0,"num_object_copies":17366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11300","reported_seq":11439,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.383324+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:05.383324+0000","last_peered":"2024-04-22T08:32:05.383324+0000","last_clean":"2024-04-22T08:32:05.383324+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:05.383324+0000","last_undegraded":"2024-04-22T08:32:05.383324+0000","last_fullsized":"2024-04-22T08:32:05.383324+0000","mapping_epoch":27,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35303424,"num_objects":8619,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11332","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.415405+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:05.415405+0000","last_peered":"2024-04-22T08:32:05.415405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:05.415405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35696640,"num_objects":8715,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8201,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4974},{"shards":"0,2","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11360","reported_seq":19096,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:09.210241+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:09.210241+0000","last_peered":"2024-04-22T08:32:09.210241+0000","last_clean":"2024-04-22T08:32:09.210241+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:09.210241+0000","last_undegraded":"2024-04-22T08:32:09.210241+0000","last_fullsized":"2024-04-22T08:32:09.210241+0000","mapping_epoch":36,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34832384,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7728,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11198","reported_seq":11343,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:05.361086+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:05.361086+0000","last_peered":"2024-04-22T08:32:05.361086+0000","last_clean":"2024-04-22T08:32:05.361086+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:05.361086+0000","last_undegraded":"2024-04-22T08:32:05.361086+0000","last_fullsized":"2024-04-22T08:32:05.361086+0000","mapping_epoch":27,"log_start":"28'1100","ondisk_log_start":"28'1100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":8608,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11494","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.596384+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:05.596384+0000","last_peered":"2024-04-22T08:32:05.596384+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:05.596384+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35747026,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2537},{"shards":"2","objects":4890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11383","reported_seq":11717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.368125+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:05.368125+0000","last_peered":"2024-04-22T08:32:05.368125+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:05.368125+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36058322,"num_objects":8805,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5162},{"shards":"0,2","objects":2485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11266","reported_seq":11567,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.549620+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:05.549620+0000","last_peered":"2024-04-22T08:32:05.549620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:05.549620+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2451},{"shards":"2","objects":4990}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11264","reported_seq":17716,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:05.569979+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:05.569979+0000","last_peered":"2024-04-22T08:32:05.569979+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:05.569979+0000","last_undegraded":"2024-04-22T08:32:05.569979+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513554,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3200,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":8065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11340","reported_seq":11682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.394315+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:05.394315+0000","last_peered":"2024-04-22T08:32:05.394315+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:05.394315+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36053609,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11240","reported_seq":11601,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.412138+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:05.412138+0000","last_peered":"2024-04-22T08:32:05.412138+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:05.412138+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1200","ondisk_log_start":"28'1200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34856960,"num_objects":8510,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7471,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4766},{"shards":"0,2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11316","reported_seq":11565,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.438173+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:05.438173+0000","last_peered":"2024-04-22T08:32:05.438173+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:05.438173+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35726546,"num_objects":8724,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5105},{"shards":"0,2","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11334","reported_seq":11696,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:05.474345+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:05.474345+0000","last_peered":"2024-04-22T08:32:05.474345+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:05.474345+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35506596,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":568500077,"num_objects":138806,"num_object_clones":0,"num_object_copies":277612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37004,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":138806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154898,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7996,"num_bytes_recovered":28594793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220374528,"data_stored":1220293325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117057,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705984,"kb_used_data":399544,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93665856,"statfs":{"total":96636764160,"available":95913836544,"internally_reserved":0,"allocated":409133056,"data_stored":408917574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.9670000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920820,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618044,"kb_used_data":315700,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93753796,"statfs":{"total":96636764160,"available":96003887104,"internally_reserved":0,"allocated":323276800,"data_stored":322998696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.4780000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313285,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825728,"kb_used_data":479160,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93546112,"statfs":{"total":96636764160,"available":95791218688,"internally_reserved":0,"allocated":490659840,"data_stored":490454892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":354873043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,1,0,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":3.585},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":3.6230000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":489123840,"data_stored":489120361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322961408,"data_stored":322891235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408289280,"data_stored":408281729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:13.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:14.728 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:14.730 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:14.777 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-04-22T08:32:14.264926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561057645,"num_objects":136989,"num_object_clones":0,"num_object_copies":273978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36900,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":136989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156715,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":29713001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131760,"kb_used_data":1184600,"kb_used_omap":12,"kb_used_meta":947123,"kb_avail":280983760,"statfs":{"total":289910292480,"available":287727370240,"internally_reserved":0,"allocated":1213030400,"data_stored":1212331866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":969854062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,4,5,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16060416,"num_objects":-3921,"num_object_clones":0,"num_object_copies":-7842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":-514,"num_objects_unfound":0,"num_objects_dirty":-3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014631"},"pg_stats":[{"pgid":"2.f","version":"37'11547","reported_seq":11883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.321040+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:10.321040+0000","last_peered":"2024-04-22T08:32:10.321040+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:10.321040+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35311616,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4963},{"shards":"0,2","objects":2331}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11603","reported_seq":11719,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.588148+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:10.588148+0000","last_peered":"2024-04-22T08:32:10.588148+0000","last_clean":"2024-04-22T08:32:10.588148+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:10.588148+0000","last_undegraded":"2024-04-22T08:32:10.588148+0000","last_fullsized":"2024-04-22T08:32:10.588148+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35344384,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8659,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11370","reported_seq":11613,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.546957+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:10.546957+0000","last_peered":"2024-04-22T08:32:10.546957+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:10.546957+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34787352,"num_objects":8494,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8197,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2319},{"shards":"2","objects":4798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11557","reported_seq":11588,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.577051+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:10.577051+0000","last_peered":"2024-04-22T08:32:10.577051+0000","last_clean":"2024-04-22T08:32:10.577051+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:10.577051+0000","last_undegraded":"2024-04-22T08:32:10.577051+0000","last_fullsized":"2024-04-22T08:32:10.577051+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35049472,"num_objects":8557,"num_object_clones":0,"num_object_copies":17114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11557,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11414","reported_seq":11553,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.466227+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:10.466227+0000","last_peered":"2024-04-22T08:32:10.466227+0000","last_clean":"2024-04-22T08:32:10.466227+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:10.466227+0000","last_undegraded":"2024-04-22T08:32:10.466227+0000","last_fullsized":"2024-04-22T08:32:10.466227+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34836480,"num_objects":8505,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11466","reported_seq":11941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.437966+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:10.437966+0000","last_peered":"2024-04-22T08:32:10.437966+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:10.437966+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35147776,"num_objects":8581,"num_object_clones":0,"num_object_copies":17162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4858},{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11360","reported_seq":19096,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:09.210241+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:09.210241+0000","last_peered":"2024-04-22T08:32:09.210241+0000","last_clean":"2024-04-22T08:32:09.210241+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:09.210241+0000","last_undegraded":"2024-04-22T08:32:09.210241+0000","last_fullsized":"2024-04-22T08:32:09.210241+0000","mapping_epoch":36,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34832384,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7728,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11326","reported_seq":11471,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.435406+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:10.435406+0000","last_peered":"2024-04-22T08:32:10.435406+0000","last_clean":"2024-04-22T08:32:10.435406+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:10.435406+0000","last_undegraded":"2024-04-22T08:32:10.435406+0000","last_fullsized":"2024-04-22T08:32:10.435406+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11631","reported_seq":11861,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.499420+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:10.499420+0000","last_peered":"2024-04-22T08:32:10.499420+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:10.499420+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35185874,"num_objects":8592,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2400},{"shards":"2","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11524","reported_seq":11903,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.447433+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:10.447433+0000","last_peered":"2024-04-22T08:32:10.447433+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:10.447433+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35480786,"num_objects":8664,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5044},{"shards":"0,2","objects":2344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11376","reported_seq":11683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.588346+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:10.588346+0000","last_peered":"2024-04-22T08:32:10.588346+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:10.588346+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35033088,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2341},{"shards":"2","objects":4883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11377","reported_seq":18376,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:10.608526+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:10.608526+0000","last_peered":"2024-04-22T08:32:10.608526+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:10.608526+0000","last_undegraded":"2024-04-22T08:32:10.608526+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050706,"num_objects":8559,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3473,"num_bytes_recovered":14209024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":334},{"shards":"2","objects":8225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11447","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.354478+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:10.354478+0000","last_peered":"2024-04-22T08:32:10.354478+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:10.354478+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615337,"num_objects":8696,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5115},{"shards":"0,2","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11348","reported_seq":11728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.404255+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:10.404255+0000","last_peered":"2024-04-22T08:32:10.404255+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:10.404255+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34414592,"num_objects":8402,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4667},{"shards":"0,2","objects":2368}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11433","reported_seq":11700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.293659+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:10.293659+0000","last_peered":"2024-04-22T08:32:10.293659+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:10.293659+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35247314,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4997},{"shards":"0,2","objects":2293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11461","reported_seq":11860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.443651+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:10.443651+0000","last_peered":"2024-04-22T08:32:10.443651+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:10.443651+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34986404,"num_objects":8545,"num_object_clones":0,"num_object_copies":17090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11458,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4791},{"shards":"0,2","objects":2427}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":561057645,"num_objects":136989,"num_object_clones":0,"num_object_copies":273978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36900,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":136989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156715,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":29713001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1210454016,"data_stored":1210372813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117058,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702268,"kb_used_data":395828,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93669572,"statfs":{"total":96636764160,"available":95917641728,"internally_reserved":0,"allocated":405327872,"data_stored":405112390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":1.786}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":1.804}]}]},{"osd":1,"up_from":33,"seq":141733920820,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618044,"kb_used_data":315700,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93753796,"statfs":{"total":96636764160,"available":96003887104,"internally_reserved":0,"allocated":323276800,"data_stored":322998696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.4780000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313286,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811448,"kb_used_data":473072,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93560392,"statfs":{"total":96636764160,"available":95805841408,"internally_reserved":0,"allocated":484425728,"data_stored":484220780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":7.9619999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":8},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":483098624,"data_stored":483095145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322961408,"data_stored":322891235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404393984,"data_stored":404386433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:14.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:15.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:15.493 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:15.534 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-04-22T08:32:14.264926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561057645,"num_objects":136989,"num_object_clones":0,"num_object_copies":273978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36900,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":136989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156715,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":29713001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131760,"kb_used_data":1184600,"kb_used_omap":12,"kb_used_meta":947123,"kb_avail":280983760,"statfs":{"total":289910292480,"available":287727370240,"internally_reserved":0,"allocated":1213030400,"data_stored":1212331866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":969854062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,4,5,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16060416,"num_objects":-3921,"num_object_clones":0,"num_object_copies":-7842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":-514,"num_objects_unfound":0,"num_objects_dirty":-3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014631"},"pg_stats":[{"pgid":"2.f","version":"37'11547","reported_seq":11883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.321040+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:10.321040+0000","last_peered":"2024-04-22T08:32:10.321040+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:10.321040+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35311616,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4963},{"shards":"0,2","objects":2331}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11603","reported_seq":11719,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.588148+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:10.588148+0000","last_peered":"2024-04-22T08:32:10.588148+0000","last_clean":"2024-04-22T08:32:10.588148+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:10.588148+0000","last_undegraded":"2024-04-22T08:32:10.588148+0000","last_fullsized":"2024-04-22T08:32:10.588148+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35344384,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8659,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11370","reported_seq":11613,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.546957+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:10.546957+0000","last_peered":"2024-04-22T08:32:10.546957+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:10.546957+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34787352,"num_objects":8494,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8197,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2319},{"shards":"2","objects":4798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11557","reported_seq":11588,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.577051+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:10.577051+0000","last_peered":"2024-04-22T08:32:10.577051+0000","last_clean":"2024-04-22T08:32:10.577051+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:10.577051+0000","last_undegraded":"2024-04-22T08:32:10.577051+0000","last_fullsized":"2024-04-22T08:32:10.577051+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35049472,"num_objects":8557,"num_object_clones":0,"num_object_copies":17114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11557,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11414","reported_seq":11553,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.466227+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:10.466227+0000","last_peered":"2024-04-22T08:32:10.466227+0000","last_clean":"2024-04-22T08:32:10.466227+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:10.466227+0000","last_undegraded":"2024-04-22T08:32:10.466227+0000","last_fullsized":"2024-04-22T08:32:10.466227+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34836480,"num_objects":8505,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11466","reported_seq":11941,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.437966+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:10.437966+0000","last_peered":"2024-04-22T08:32:10.437966+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:10.437966+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35147776,"num_objects":8581,"num_object_clones":0,"num_object_copies":17162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4858},{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11360","reported_seq":19096,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:09.210241+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:09.210241+0000","last_peered":"2024-04-22T08:32:09.210241+0000","last_clean":"2024-04-22T08:32:09.210241+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:09.210241+0000","last_undegraded":"2024-04-22T08:32:09.210241+0000","last_fullsized":"2024-04-22T08:32:09.210241+0000","mapping_epoch":36,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34832384,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7728,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11326","reported_seq":11471,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:10.435406+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:10.435406+0000","last_peered":"2024-04-22T08:32:10.435406+0000","last_clean":"2024-04-22T08:32:10.435406+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:10.435406+0000","last_undegraded":"2024-04-22T08:32:10.435406+0000","last_fullsized":"2024-04-22T08:32:10.435406+0000","mapping_epoch":27,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11631","reported_seq":11861,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.499420+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:10.499420+0000","last_peered":"2024-04-22T08:32:10.499420+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:10.499420+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35185874,"num_objects":8592,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2400},{"shards":"2","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11524","reported_seq":11903,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.447433+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:10.447433+0000","last_peered":"2024-04-22T08:32:10.447433+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:10.447433+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35480786,"num_objects":8664,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5044},{"shards":"0,2","objects":2344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11376","reported_seq":11683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.588346+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:10.588346+0000","last_peered":"2024-04-22T08:32:10.588346+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:10.588346+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35033088,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2341},{"shards":"2","objects":4883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11377","reported_seq":18376,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:10.608526+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:10.608526+0000","last_peered":"2024-04-22T08:32:10.608526+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:10.608526+0000","last_undegraded":"2024-04-22T08:32:10.608526+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050706,"num_objects":8559,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3473,"num_bytes_recovered":14209024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":334},{"shards":"2","objects":8225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11447","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.354478+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:10.354478+0000","last_peered":"2024-04-22T08:32:10.354478+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:10.354478+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615337,"num_objects":8696,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5115},{"shards":"0,2","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11348","reported_seq":11728,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.404255+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:10.404255+0000","last_peered":"2024-04-22T08:32:10.404255+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:10.404255+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1300","ondisk_log_start":"28'1300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34414592,"num_objects":8402,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7579,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4667},{"shards":"0,2","objects":2368}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11433","reported_seq":11700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.293659+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:10.293659+0000","last_peered":"2024-04-22T08:32:10.293659+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:10.293659+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35247314,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4997},{"shards":"0,2","objects":2293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-22T08:30:09.143826+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:09.143826+0000","last_peered":"2024-04-22T08:30:09.143826+0000","last_clean":"2024-04-22T08:30:09.143826+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:09.143826+0000","last_undegraded":"2024-04-22T08:30:09.143826+0000","last_fullsized":"2024-04-22T08:30:09.143826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11461","reported_seq":11860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:10.443651+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:10.443651+0000","last_peered":"2024-04-22T08:32:10.443651+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:10.443651+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34986404,"num_objects":8545,"num_object_clones":0,"num_object_copies":17090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11458,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4791},{"shards":"0,2","objects":2427}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":561057645,"num_objects":136989,"num_object_clones":0,"num_object_copies":273978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36900,"num_objects_misplaced":334,"num_objects_unfound":0,"num_objects_dirty":136989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156715,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8373,"num_bytes_recovered":29713001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1210454016,"data_stored":1210372813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117058,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702268,"kb_used_data":395828,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93669572,"statfs":{"total":96636764160,"available":95917641728,"internally_reserved":0,"allocated":405327872,"data_stored":405112390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.105,"15min":0.89100000000000001},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":13.635,"5min":13.635,"15min":13.635},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.22,"5min":0.99299999999999999,"15min":0.68700000000000006},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":13.721,"5min":13.721,"15min":13.721},"last":1.786}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":1.804}]}]},{"osd":1,"up_from":33,"seq":141733920820,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618044,"kb_used_data":315700,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93753796,"statfs":{"total":96636764160,"available":96003887104,"internally_reserved":0,"allocated":323276800,"data_stored":322998696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.4780000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313286,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811448,"kb_used_data":473072,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93560392,"statfs":{"total":96636764160,"available":95805841408,"internally_reserved":0,"allocated":484425728,"data_stored":484220780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":346484435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":7.9619999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.86599999999999999,"15min":0.749},"min":{"1min":0.221,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.754,"5min":6.9580000000000002,"15min":6.9580000000000002},"last":8},{"interface":"front","average":{"1min":0.879,"5min":0.85899999999999999,"15min":0.80500000000000005},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":483098624,"data_stored":483095145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322961408,"data_stored":322891235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404393984,"data_stored":404386433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'11547", 'reported_seq': 11883, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.321040+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:10.321040+0000', 'last_peered': '2024-04-22T08:32:10.321040+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:10.321040+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35311616, 'num_objects': 8621, 'num_object_clones': 0, 'num_object_copies': 17242, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3658, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 136, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4963}, {'shards': '0,2', 'objects': 2331}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'11370", 'reported_seq': 11613, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.546957+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:10.546957+0000', 'last_peered': '2024-04-22T08:32:10.546957+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:10.546957+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34787352, 'num_objects': 8494, 'num_object_clones': 0, 'num_object_copies': 16988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3697, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8494, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8197, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2319}, {'shards': '2', 'objects': 4798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:15.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'11466", 'reported_seq': 11941, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.437966+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:32:10.437966+0000', 'last_peered': '2024-04-22T08:32:10.437966+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:10.437966+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35147776, 'num_objects': 8581, 'num_object_clones': 0, 'num_object_copies': 17162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8335, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 157, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4858}, {'shards': '0,2', 'objects': 2439}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'11631", 'reported_seq': 11861, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.499420+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:10.499420+0000', 'last_peered': '2024-04-22T08:32:10.499420+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:10.499420+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35185874, 'num_objects': 8592, 'num_object_clones': 0, 'num_object_copies': 17184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3837, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11571, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2400}, {'shards': '2', 'objects': 4757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'11524", 'reported_seq': 11903, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.447433+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:10.447433+0000', 'last_peered': '2024-04-22T08:32:10.447433+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:10.447433+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35480786, 'num_objects': 8664, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3622, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11522, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 154, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5044}, {'shards': '0,2', 'objects': 2344}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'11376", 'reported_seq': 11683, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.588346+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:10.588346+0000', 'last_peered': '2024-04-22T08:32:10.588346+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:10.588346+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35033088, 'num_objects': 8553, 'num_object_clones': 0, 'num_object_copies': 17106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3672, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7830, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2341}, {'shards': '2', 'objects': 4883}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'11377", 'reported_seq': 18376, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:32:10.608526+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:32:10.608526+0000', 'last_peered': '2024-04-22T08:32:10.608526+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:32:10.608526+0000', 'last_undegraded': '2024-04-22T08:32:10.608526+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35050706, 'num_objects': 8559, 'num_object_clones': 0, 'num_object_copies': 17118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 334, 'num_objects_unfound': 0, 'num_objects_dirty': 8559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11375, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3473, 'num_bytes_recovered': 14209024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 334}, {'shards': '2', 'objects': 8225}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:15.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'11447", 'reported_seq': 11807, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.354478+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:10.354478+0000', 'last_peered': '2024-04-22T08:32:10.354478+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:10.354478+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35615337, 'num_objects': 8696, 'num_object_clones': 0, 'num_object_copies': 17392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11446, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 142, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5115}, {'shards': '0,2', 'objects': 2346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'11348", 'reported_seq': 11728, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.404255+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:10.404255+0000', 'last_peered': '2024-04-22T08:32:10.404255+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:10.404255+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34414592, 'num_objects': 8402, 'num_object_clones': 0, 'num_object_copies': 16804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3738, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7579, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4667}, {'shards': '0,2', 'objects': 2368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'11433", 'reported_seq': 11700, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.293659+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:10.293659+0000', 'last_peered': '2024-04-22T08:32:10.293659+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:10.293659+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35247314, 'num_objects': 8607, 'num_object_clones': 0, 'num_object_copies': 17214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3612, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11431, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4997}, {'shards': '0,2', 'objects': 2293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:15.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'11461", 'reported_seq': 11860, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:10.443651+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:10.443651+0000', 'last_peered': '2024-04-22T08:32:10.443651+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:10.443651+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34986404, 'num_objects': 8545, 'num_object_clones': 0, 'num_object_copies': 17090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3757, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11458, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 132, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4791}, {'shards': '0,2', 'objects': 2427}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:15.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:16.321 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36900/273978 objects degraded (13.468%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":136989,"data_bytes":561057645,"bytes_used":2182922240,"bytes_avail":287727370240,"bytes_total":289910292480,"degraded_objects":36900,"degraded_total":273978,"degraded_ratio":0.13468234675776888,"misplaced_objects":334,"misplaced_total":273978,"misplaced_ratio":0.0012190759842031112,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":175231,"recovering_keys_per_sec":0,"num_objects_recovered":702,"num_bytes_recovered":2105344,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:16.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:16.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:17.247 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:17.248 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:17.310 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-22T08:32:16.265384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552505197,"num_objects":134901,"num_object_clones":0,"num_object_copies":269802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158803,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2155536,"kb_used_data":1173944,"kb_used_omap":12,"kb_used_meta":981555,"kb_avail":280959984,"statfs":{"total":289910292480,"available":287703023616,"internally_reserved":0,"allocated":1202118656,"data_stored":1201420122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1005112430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,6,1,5,8,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24076288,"num_objects":-5878,"num_object_clones":0,"num_object_copies":-11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":-754,"num_objects_unfound":0,"num_objects_dirty":-5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1021,"num_bytes_recovered":3088384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014510"},"pg_stats":[{"pgid":"2.f","version":"37'11669","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.470646+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:15.470646+0000","last_peered":"2024-04-22T08:32:15.470646+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:15.470646+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811904,"num_objects":8499,"num_object_clones":0,"num_object_copies":16998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4848},{"shards":"0,2","objects":2209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11740","reported_seq":11856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.548403+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:15.548403+0000","last_peered":"2024-04-22T08:32:15.548403+0000","last_clean":"2024-04-22T08:32:15.548403+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:15.548403+0000","last_undegraded":"2024-04-22T08:32:15.548403+0000","last_fullsized":"2024-04-22T08:32:15.548403+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34783232,"num_objects":8492,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11520","reported_seq":11783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.559597+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:15.559597+0000","last_peered":"2024-04-22T08:32:15.559597+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:15.559597+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34172952,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8347,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2169},{"shards":"2","objects":4658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11683","reported_seq":11714,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.523696+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:15.523696+0000","last_peered":"2024-04-22T08:32:15.523696+0000","last_clean":"2024-04-22T08:32:15.523696+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:15.523696+0000","last_undegraded":"2024-04-22T08:32:15.523696+0000","last_fullsized":"2024-04-22T08:32:15.523696+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533376,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11683,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11530","reported_seq":11669,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.491540+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:15.491540+0000","last_peered":"2024-04-22T08:32:15.491540+0000","last_clean":"2024-04-22T08:32:15.491540+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:15.491540+0000","last_undegraded":"2024-04-22T08:32:15.491540+0000","last_fullsized":"2024-04-22T08:32:15.491540+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34361344,"num_objects":8389,"num_object_clones":0,"num_object_copies":16778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11609","reported_seq":12100,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.378583+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:15.378583+0000","last_peered":"2024-04-22T08:32:15.378583+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:15.378583+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34562048,"num_objects":8438,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4723},{"shards":"0,2","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'11490","reported_seq":19226,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:14.184016+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:14.184016+0000","last_peered":"2024-04-22T08:32:14.184016+0000","last_clean":"2024-04-22T08:32:14.184016+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:14.184016+0000","last_undegraded":"2024-04-22T08:32:14.184016+0000","last_fullsized":"2024-04-22T08:32:14.184016+0000","mapping_epoch":36,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34299904,"num_objects":8374,"num_object_clones":0,"num_object_copies":16748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11451","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.335610+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:15.335610+0000","last_peered":"2024-04-22T08:32:15.335610+0000","last_clean":"2024-04-22T08:32:15.335610+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:15.335610+0000","last_undegraded":"2024-04-22T08:32:15.335610+0000","last_fullsized":"2024-04-22T08:32:15.335610+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34222080,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141426+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:30:10.141426+0000","last_peered":"2024-04-22T08:30:10.141426+0000","last_clean":"2024-04-22T08:30:10.141426+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:30:10.141426+0000","last_undegraded":"2024-04-22T08:30:10.141426+0000","last_fullsized":"2024-04-22T08:30:10.141426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11768","reported_seq":12008,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.581431+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:15.581431+0000","last_peered":"2024-04-22T08:32:15.581431+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:15.581431+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624722,"num_objects":8455,"num_object_clones":0,"num_object_copies":16910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2263},{"shards":"2","objects":4625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11647","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.405731+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:15.405731+0000","last_peered":"2024-04-22T08:32:15.405731+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:15.405731+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976978,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4931},{"shards":"0,2","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11489","reported_seq":11806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.575890+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:15.575890+0000","last_peered":"2024-04-22T08:32:15.575890+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:15.575890+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34570240,"num_objects":8440,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2228},{"shards":"2","objects":4775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141578+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:30:10.141578+0000","last_peered":"2024-04-22T08:30:10.141578+0000","last_clean":"2024-04-22T08:30:10.141578+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:30:10.141578+0000","last_undegraded":"2024-04-22T08:30:10.141578+0000","last_fullsized":"2024-04-22T08:30:10.141578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11522","reported_seq":19001,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:15.618015+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:15.618015+0000","last_peered":"2024-04-22T08:32:15.618015+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:15.618015+0000","last_undegraded":"2024-04-22T08:32:15.618015+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34456786,"num_objects":8414,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":8414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3713,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":94},{"shards":"2","objects":8320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11595","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.348724+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:15.348724+0000","last_peered":"2024-04-22T08:32:15.348724+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:15.348724+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009129,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4976},{"shards":"0,2","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.142371+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:30:10.142371+0000","last_peered":"2024-04-22T08:30:10.142371+0000","last_clean":"2024-04-22T08:30:10.142371+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:30:10.142371+0000","last_undegraded":"2024-04-22T08:30:10.142371+0000","last_fullsized":"2024-04-22T08:30:10.142371+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11469","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.469118+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:15.469118+0000","last_peered":"2024-04-22T08:32:15.469118+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:15.469118+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33918976,"num_objects":8281,"num_object_clones":0,"num_object_copies":16562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11561","reported_seq":11840,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.415098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:15.415098+0000","last_peered":"2024-04-22T08:32:15.415098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:15.415098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34723026,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4875},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.141628+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:30:10.141628+0000","last_peered":"2024-04-22T08:30:10.141628+0000","last_clean":"2024-04-22T08:30:10.141628+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:30:10.141628+0000","last_undegraded":"2024-04-22T08:30:10.141628+0000","last_fullsized":"2024-04-22T08:30:10.141628+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.144020+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:30:10.144020+0000","last_peered":"2024-04-22T08:30:10.144020+0000","last_clean":"2024-04-22T08:30:10.144020+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:30:10.144020+0000","last_undegraded":"2024-04-22T08:30:10.144020+0000","last_fullsized":"2024-04-22T08:30:10.144020+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11585","reported_seq":12006,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.387195+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:15.387195+0000","last_peered":"2024-04-22T08:32:15.387195+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:15.387195+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34478500,"num_objects":8421,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4678},{"shards":"0,2","objects":2303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":552505197,"num_objects":134901,"num_object_clones":0,"num_object_copies":269802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158803,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1199669248,"data_stored":1199588045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117059,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702148,"kb_used_data":391612,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93669692,"statfs":{"total":96636764160,"available":95917764608,"internally_reserved":0,"allocated":401010688,"data_stored":400795206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,3,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23300000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920821,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626096,"kb_used_data":315560,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93745744,"statfs":{"total":96636764160,"available":95995641856,"internally_reserved":0,"allocated":323133440,"data_stored":322855336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":317976268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.4780000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313287,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827292,"kb_used_data":466772,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93544548,"statfs":{"total":96636764160,"available":95789617152,"internally_reserved":0,"allocated":477974528,"data_stored":477769580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.9500000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":4.6980000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476778496,"data_stored":476775017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322818048,"data_stored":322747875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":400072704,"data_stored":400065153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:20.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:21.013 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:21.014 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:21.093 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-04-22T08:32:20.266351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552013677,"num_objects":134781,"num_object_clones":0,"num_object_copies":269562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158923,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2147108,"kb_used_data":1173708,"kb_used_omap":12,"kb_used_meta":973363,"kb_avail":280968412,"statfs":{"total":289910292480,"available":287711653888,"internally_reserved":0,"allocated":1201876992,"data_stored":1201180204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":996723822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,5,2,7,8,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9043968,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006033"},"pg_stats":[{"pgid":"2.f","version":"37'11669","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.470646+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:15.470646+0000","last_peered":"2024-04-22T08:32:15.470646+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:15.470646+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811904,"num_objects":8499,"num_object_clones":0,"num_object_copies":16998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4848},{"shards":"0,2","objects":2209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11740","reported_seq":11856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.548403+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:15.548403+0000","last_peered":"2024-04-22T08:32:15.548403+0000","last_clean":"2024-04-22T08:32:15.548403+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:15.548403+0000","last_undegraded":"2024-04-22T08:32:15.548403+0000","last_fullsized":"2024-04-22T08:32:15.548403+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34783232,"num_objects":8492,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11520","reported_seq":11783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.559597+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:15.559597+0000","last_peered":"2024-04-22T08:32:15.559597+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:15.559597+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34172952,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8347,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2169},{"shards":"2","objects":4658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11683","reported_seq":11714,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.523696+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:15.523696+0000","last_peered":"2024-04-22T08:32:15.523696+0000","last_clean":"2024-04-22T08:32:15.523696+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:15.523696+0000","last_undegraded":"2024-04-22T08:32:15.523696+0000","last_fullsized":"2024-04-22T08:32:15.523696+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533376,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11683,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11530","reported_seq":11669,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.491540+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:15.491540+0000","last_peered":"2024-04-22T08:32:15.491540+0000","last_clean":"2024-04-22T08:32:15.491540+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:15.491540+0000","last_undegraded":"2024-04-22T08:32:15.491540+0000","last_fullsized":"2024-04-22T08:32:15.491540+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34361344,"num_objects":8389,"num_object_clones":0,"num_object_copies":16778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11609","reported_seq":12100,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.378583+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:15.378583+0000","last_peered":"2024-04-22T08:32:15.378583+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:15.378583+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34562048,"num_objects":8438,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4723},{"shards":"0,2","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11610","reported_seq":19348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.242834+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:19.242834+0000","last_peered":"2024-04-22T08:32:19.242834+0000","last_clean":"2024-04-22T08:32:19.242834+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:19.242834+0000","last_undegraded":"2024-04-22T08:32:19.242834+0000","last_fullsized":"2024-04-22T08:32:19.242834+0000","mapping_epoch":36,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11451","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.335610+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:15.335610+0000","last_peered":"2024-04-22T08:32:15.335610+0000","last_clean":"2024-04-22T08:32:15.335610+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:15.335610+0000","last_undegraded":"2024-04-22T08:32:15.335610+0000","last_fullsized":"2024-04-22T08:32:15.335610+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34222080,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11768","reported_seq":12008,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.581431+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:15.581431+0000","last_peered":"2024-04-22T08:32:15.581431+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:15.581431+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624722,"num_objects":8455,"num_object_clones":0,"num_object_copies":16910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2263},{"shards":"2","objects":4625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11647","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.405731+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:15.405731+0000","last_peered":"2024-04-22T08:32:15.405731+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:15.405731+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976978,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4931},{"shards":"0,2","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11489","reported_seq":11806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.575890+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:15.575890+0000","last_peered":"2024-04-22T08:32:15.575890+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:15.575890+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34570240,"num_objects":8440,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2228},{"shards":"2","objects":4775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11522","reported_seq":19001,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:15.618015+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:15.618015+0000","last_peered":"2024-04-22T08:32:15.618015+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:15.618015+0000","last_undegraded":"2024-04-22T08:32:15.618015+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34456786,"num_objects":8414,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":8414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3713,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":94},{"shards":"2","objects":8320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11595","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.348724+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:15.348724+0000","last_peered":"2024-04-22T08:32:15.348724+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:15.348724+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009129,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4976},{"shards":"0,2","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11469","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.469118+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:15.469118+0000","last_peered":"2024-04-22T08:32:15.469118+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:15.469118+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33918976,"num_objects":8281,"num_object_clones":0,"num_object_copies":16562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11561","reported_seq":11840,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.415098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:15.415098+0000","last_peered":"2024-04-22T08:32:15.415098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:15.415098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34723026,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4875},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11585","reported_seq":12006,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.387195+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:15.387195+0000","last_peered":"2024-04-22T08:32:15.387195+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:15.387195+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34478500,"num_objects":8421,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4678},{"shards":"0,2","objects":2303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":552013677,"num_objects":134781,"num_object_clones":0,"num_object_copies":269562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158923,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1199353856,"data_stored":1199279611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117059,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702148,"kb_used_data":391612,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93669692,"statfs":{"total":96636764160,"available":95917764608,"internally_reserved":0,"allocated":401010688,"data_stored":400795206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,3,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23300000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920822,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617668,"kb_used_data":315324,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93754172,"statfs":{"total":96636764160,"available":96004272128,"internally_reserved":0,"allocated":322891776,"data_stored":322615418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.88}]}]},{"osd":0,"up_from":21,"seq":90194313287,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827292,"kb_used_data":466772,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93544548,"statfs":{"total":96636764160,"available":95789617152,"internally_reserved":0,"allocated":477974528,"data_stored":477769580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.9500000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":4.6980000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476778496,"data_stored":476775017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322502656,"data_stored":322439441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":400072704,"data_stored":400065153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:21.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:21.786 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:21.846 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-04-22T08:32:20.266351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552013677,"num_objects":134781,"num_object_clones":0,"num_object_copies":269562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158923,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2147108,"kb_used_data":1173708,"kb_used_omap":12,"kb_used_meta":973363,"kb_avail":280968412,"statfs":{"total":289910292480,"available":287711653888,"internally_reserved":0,"allocated":1201876992,"data_stored":1201180204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":996723822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,5,2,7,8,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9043968,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006033"},"pg_stats":[{"pgid":"2.f","version":"37'11669","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.470646+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:15.470646+0000","last_peered":"2024-04-22T08:32:15.470646+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:15.470646+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811904,"num_objects":8499,"num_object_clones":0,"num_object_copies":16998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4848},{"shards":"0,2","objects":2209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'11740","reported_seq":11856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.548403+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:15.548403+0000","last_peered":"2024-04-22T08:32:15.548403+0000","last_clean":"2024-04-22T08:32:15.548403+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:15.548403+0000","last_undegraded":"2024-04-22T08:32:15.548403+0000","last_fullsized":"2024-04-22T08:32:15.548403+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34783232,"num_objects":8492,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'11520","reported_seq":11783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.559597+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:15.559597+0000","last_peered":"2024-04-22T08:32:15.559597+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:15.559597+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34172952,"num_objects":8344,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8347,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2169},{"shards":"2","objects":4658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'11683","reported_seq":11714,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.523696+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:15.523696+0000","last_peered":"2024-04-22T08:32:15.523696+0000","last_clean":"2024-04-22T08:32:15.523696+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:15.523696+0000","last_undegraded":"2024-04-22T08:32:15.523696+0000","last_fullsized":"2024-04-22T08:32:15.523696+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533376,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11683,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'11530","reported_seq":11669,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.491540+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:15.491540+0000","last_peered":"2024-04-22T08:32:15.491540+0000","last_clean":"2024-04-22T08:32:15.491540+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:15.491540+0000","last_undegraded":"2024-04-22T08:32:15.491540+0000","last_fullsized":"2024-04-22T08:32:15.491540+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34361344,"num_objects":8389,"num_object_clones":0,"num_object_copies":16778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'11609","reported_seq":12100,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.378583+0000","last_change":"2024-04-22T08:27:56.596202+0000","last_active":"2024-04-22T08:32:15.378583+0000","last_peered":"2024-04-22T08:32:15.378583+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:15.378583+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34562048,"num_objects":8438,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4723},{"shards":"0,2","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11610","reported_seq":19348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.242834+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:19.242834+0000","last_peered":"2024-04-22T08:32:19.242834+0000","last_clean":"2024-04-22T08:32:19.242834+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:19.242834+0000","last_undegraded":"2024-04-22T08:32:19.242834+0000","last_fullsized":"2024-04-22T08:32:19.242834+0000","mapping_epoch":36,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"37'11451","reported_seq":11596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:32:15.335610+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:15.335610+0000","last_peered":"2024-04-22T08:32:15.335610+0000","last_clean":"2024-04-22T08:32:15.335610+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:15.335610+0000","last_undegraded":"2024-04-22T08:32:15.335610+0000","last_fullsized":"2024-04-22T08:32:15.335610+0000","mapping_epoch":27,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34222080,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'11768","reported_seq":12008,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.581431+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:15.581431+0000","last_peered":"2024-04-22T08:32:15.581431+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:15.581431+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624722,"num_objects":8455,"num_object_clones":0,"num_object_copies":16910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2263},{"shards":"2","objects":4625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.165615+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:30:10.165615+0000","last_peered":"2024-04-22T08:30:10.165615+0000","last_clean":"2024-04-22T08:30:10.165615+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:30:10.165615+0000","last_undegraded":"2024-04-22T08:30:10.165615+0000","last_fullsized":"2024-04-22T08:30:10.165615+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'11647","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.405731+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:15.405731+0000","last_peered":"2024-04-22T08:32:15.405731+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:15.405731+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976978,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4931},{"shards":"0,2","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":68,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.145333+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:30:10.145333+0000","last_peered":"2024-04-22T08:30:10.145333+0000","last_clean":"2024-04-22T08:30:10.145333+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:30:10.145333+0000","last_undegraded":"2024-04-22T08:30:10.145333+0000","last_fullsized":"2024-04-22T08:30:10.145333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'11489","reported_seq":11806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.575890+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:15.575890+0000","last_peered":"2024-04-22T08:32:15.575890+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:15.575890+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34570240,"num_objects":8440,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2228},{"shards":"2","objects":4775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'11522","reported_seq":19001,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-22T08:32:15.618015+0000","last_change":"2024-04-22T08:30:08.931666+0000","last_active":"2024-04-22T08:32:15.618015+0000","last_peered":"2024-04-22T08:32:15.618015+0000","last_clean":"2024-04-22T08:27:54.074517+0000","last_became_active":"2024-04-22T08:27:56.614702+0000","last_became_peered":"2024-04-22T08:27:56.614702+0000","last_unstale":"2024-04-22T08:32:15.618015+0000","last_undegraded":"2024-04-22T08:32:15.618015+0000","last_fullsized":"2024-04-22T08:27:56.089187+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34456786,"num_objects":8414,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":8414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3713,"num_bytes_recovered":15192064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":94},{"shards":"2","objects":8320}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'11595","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.348724+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:15.348724+0000","last_peered":"2024-04-22T08:32:15.348724+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:15.348724+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009129,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4976},{"shards":"0,2","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'11469","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.469118+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:15.469118+0000","last_peered":"2024-04-22T08:32:15.469118+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:15.469118+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1400","ondisk_log_start":"28'1400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33918976,"num_objects":8281,"num_object_clones":0,"num_object_copies":16562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-22T08:30:10.170858+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:30:10.170858+0000","last_peered":"2024-04-22T08:30:10.170858+0000","last_clean":"2024-04-22T08:30:10.170858+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:30:10.170858+0000","last_undegraded":"2024-04-22T08:30:10.170858+0000","last_fullsized":"2024-04-22T08:30:10.170858+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'11561","reported_seq":11840,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.415098+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:15.415098+0000","last_peered":"2024-04-22T08:32:15.415098+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:15.415098+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34723026,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4875},{"shards":"0,2","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'11585","reported_seq":12006,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:15.387195+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:15.387195+0000","last_peered":"2024-04-22T08:32:15.387195+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:15.387195+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34478500,"num_objects":8421,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4678},{"shards":"0,2","objects":2303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":552013677,"num_objects":134781,"num_object_clones":0,"num_object_copies":269562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36821,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":134781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158923,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8692,"num_bytes_recovered":30696041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1199353856,"data_stored":1199279611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117059,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702148,"kb_used_data":391612,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93669692,"statfs":{"total":96636764160,"available":95917764608,"internally_reserved":0,"allocated":401010688,"data_stored":400795206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,3,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23300000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920822,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617668,"kb_used_data":315324,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93754172,"statfs":{"total":96636764160,"available":96004272128,"internally_reserved":0,"allocated":322891776,"data_stored":322615418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.88}]}]},{"osd":0,"up_from":21,"seq":90194313287,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":827292,"kb_used_data":466772,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93544548,"statfs":{"total":96636764160,"available":95789617152,"internally_reserved":0,"allocated":477974528,"data_stored":477769580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.9500000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":4.6980000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476778496,"data_stored":476775017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322502656,"data_stored":322439441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":400072704,"data_stored":400065153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'11669", 'reported_seq': 12019, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.470646+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:15.470646+0000', 'last_peered': '2024-04-22T08:32:15.470646+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:15.470646+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34811904, 'num_objects': 8499, 'num_object_clones': 0, 'num_object_copies': 16998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3651, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11669, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 143, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4848}, {'shards': '0,2', 'objects': 2209}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'11520", 'reported_seq': 11783, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.559597+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:15.559597+0000', 'last_peered': '2024-04-22T08:32:15.559597+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:15.559597+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34172952, 'num_objects': 8344, 'num_object_clones': 0, 'num_object_copies': 16688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3687, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8344, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8347, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2169}, {'shards': '2', 'objects': 4658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'11609", 'reported_seq': 12100, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.378583+0000', 'last_change': '2024-04-22T08:27:56.596202+0000', 'last_active': '2024-04-22T08:32:15.378583+0000', 'last_peered': '2024-04-22T08:32:15.378583+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:15.378583+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34562048, 'num_objects': 8438, 'num_object_clones': 0, 'num_object_copies': 16876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3717, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 165, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4723}, {'shards': '0,2', 'objects': 2296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:21.848 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'11768", 'reported_seq': 12008, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.581431+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:15.581431+0000', 'last_peered': '2024-04-22T08:32:15.581431+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:15.581431+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34624722, 'num_objects': 8455, 'num_object_clones': 0, 'num_object_copies': 16910, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3832, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11708, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2263}, {'shards': '2', 'objects': 4625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'11647", 'reported_seq': 12047, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.405731+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:15.405731+0000', 'last_peered': '2024-04-22T08:32:15.405731+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:15.405731+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34976978, 'num_objects': 8541, 'num_object_clones': 0, 'num_object_copies': 17082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3612, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11645, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 164, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4931}, {'shards': '0,2', 'objects': 2221}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'11489", 'reported_seq': 11806, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.575890+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:15.575890+0000', 'last_peered': '2024-04-22T08:32:15.575890+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:15.575890+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34570240, 'num_objects': 8440, 'num_object_clones': 0, 'num_object_copies': 16880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3667, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7943, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2228}, {'shards': '2', 'objects': 4775}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'11522", 'reported_seq': 19001, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-04-22T08:32:15.618015+0000', 'last_change': '2024-04-22T08:30:08.931666+0000', 'last_active': '2024-04-22T08:32:15.618015+0000', 'last_peered': '2024-04-22T08:32:15.618015+0000', 'last_clean': '2024-04-22T08:27:54.074517+0000', 'last_became_active': '2024-04-22T08:27:56.614702+0000', 'last_became_peered': '2024-04-22T08:27:56.614702+0000', 'last_unstale': '2024-04-22T08:32:15.618015+0000', 'last_undegraded': '2024-04-22T08:32:15.618015+0000', 'last_fullsized': '2024-04-22T08:27:56.089187+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34456786, 'num_objects': 8414, 'num_object_clones': 0, 'num_object_copies': 16828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 94, 'num_objects_unfound': 0, 'num_objects_dirty': 8414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11520, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3713, 'num_bytes_recovered': 15192064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 94}, {'shards': '2', 'objects': 8320}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:21.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:21.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'11595", 'reported_seq': 11973, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.348724+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:15.348724+0000', 'last_peered': '2024-04-22T08:32:15.348724+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:15.348724+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35009129, 'num_objects': 8548, 'num_object_clones': 0, 'num_object_copies': 17096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3573, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11594, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 151, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4976}, {'shards': '0,2', 'objects': 2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:21.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'11469", 'reported_seq': 11865, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.469118+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:15.469118+0000', 'last_peered': '2024-04-22T08:32:15.469118+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:15.469118+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33918976, 'num_objects': 8281, 'num_object_clones': 0, 'num_object_copies': 16562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3730, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7700, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 110, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4554}, {'shards': '0,2', 'objects': 2247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.850 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:21.851 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'11561", 'reported_seq': 11840, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.415098+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:15.415098+0000', 'last_peered': '2024-04-22T08:32:15.415098+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:15.415098+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34723026, 'num_objects': 8479, 'num_object_clones': 0, 'num_object_copies': 16958, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3606, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11559, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4875}, {'shards': '0,2', 'objects': 2165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.851 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:21.851 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'11585", 'reported_seq': 12006, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:15.387195+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:15.387195+0000', 'last_peered': '2024-04-22T08:32:15.387195+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:15.387195+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34478500, 'num_objects': 8421, 'num_object_clones': 0, 'num_object_copies': 16842, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3746, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11582, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 143, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4678}, {'shards': '0,2', 'objects': 2303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:21.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:22.581 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:22.581 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36821/269562 objects degraded (13.660%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":377,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":134781,"data_bytes":552013677,"bytes_used":2198638592,"bytes_avail":287711653888,"bytes_total":289910292480,"degraded_objects":36821,"degraded_total":269562,"degraded_ratio":0.13659566259339223,"misplaced_objects":94,"misplaced_total":269562,"misplaced_ratio":0.0003487138394877617,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":122787,"recovering_keys_per_sec":0,"num_objects_recovered":319,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":275},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-22T08:30:30.214361+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:22.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:22.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:23.245 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:23.245 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:23.296 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-22T08:32:22.266824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544096109,"num_objects":132848,"num_object_clones":0,"num_object_copies":265696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34527,"num_objects_misplaced":2169,"num_objects_unfound":0,"num_objects_dirty":132848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160856,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8911,"num_bytes_recovered":31343209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2129048,"kb_used_data":1163828,"kb_used_omap":12,"kb_used_meta":965171,"kb_avail":280986472,"statfs":{"total":289910292480,"available":287730147328,"internally_reserved":0,"allocated":1191759872,"data_stored":1191042194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":988335214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,11,7,2,1,0,3,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16961536,"num_objects":-4141,"num_object_clones":0,"num_object_copies":-8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2373,"num_objects_misplaced":1835,"num_objects_unfound":0,"num_objects_dirty":-4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4141,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001897"},"pg_stats":[{"pgid":"2.f","version":"39'11808","reported_seq":12186,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.404417+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:20.404417+0000","last_peered":"2024-04-22T08:32:20.404417+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:20.404417+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34242560,"num_objects":8360,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3638,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4722},{"shards":"0,2","objects":2070}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'11852","reported_seq":11970,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:20.582565+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:20.582565+0000","last_peered":"2024-04-22T08:32:20.582565+0000","last_clean":"2024-04-22T08:32:20.582565+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:20.582565+0000","last_undegraded":"2024-04-22T08:32:20.582565+0000","last_fullsized":"2024-04-22T08:32:20.582565+0000","mapping_epoch":27,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34324480,"num_objects":8380,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11634","reported_seq":11913,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.640173+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:20.640173+0000","last_peered":"2024-04-22T08:32:20.640173+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:20.640173+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33706008,"num_objects":8230,"num_object_clones":0,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8461,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2055},{"shards":"2","objects":4551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'11814","reported_seq":11847,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:20.610426+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:20.610426+0000","last_peered":"2024-04-22T08:32:20.610426+0000","last_clean":"2024-04-22T08:32:20.610426+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:20.610426+0000","last_undegraded":"2024-04-22T08:32:20.610426+0000","last_fullsized":"2024-04-22T08:32:20.610426+0000","mapping_epoch":27,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11664","reported_seq":11805,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:20.352800+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:20.352800+0000","last_peered":"2024-04-22T08:32:20.352800+0000","last_clean":"2024-04-22T08:32:20.352800+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:20.352800+0000","last_undegraded":"2024-04-22T08:32:20.352800+0000","last_fullsized":"2024-04-22T08:32:20.352800+0000","mapping_epoch":27,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33812480,"num_objects":8255,"num_object_clones":0,"num_object_copies":16510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'11730","reported_seq":12361,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.456156+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:20.456156+0000","last_peered":"2024-04-22T08:32:20.456156+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:20.456156+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34066432,"num_objects":8317,"num_object_clones":0,"num_object_copies":16634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":2169,"num_objects_unfound":0,"num_objects_dirty":8317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":266240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4670},{"shards":"0,2","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11610","reported_seq":19348,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.242834+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:19.242834+0000","last_peered":"2024-04-22T08:32:19.242834+0000","last_clean":"2024-04-22T08:32:19.242834+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:19.242834+0000","last_undegraded":"2024-04-22T08:32:19.242834+0000","last_fullsized":"2024-04-22T08:32:19.242834+0000","mapping_epoch":36,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11598","reported_seq":11745,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:20.480793+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:20.480793+0000","last_peered":"2024-04-22T08:32:20.480793+0000","last_clean":"2024-04-22T08:32:20.480793+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:20.480793+0000","last_undegraded":"2024-04-22T08:32:20.480793+0000","last_fullsized":"2024-04-22T08:32:20.480793+0000","mapping_epoch":27,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33619968,"num_objects":8208,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'11896","reported_seq":12146,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.566657+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:20.566657+0000","last_peered":"2024-04-22T08:32:20.566657+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:20.566657+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100434,"num_objects":8327,"num_object_clones":0,"num_object_copies":16654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2135},{"shards":"2","objects":4501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'11779","reported_seq":12186,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.480071+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:20.480071+0000","last_peered":"2024-04-22T08:32:20.480071+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:20.480071+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34436306,"num_objects":8409,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4804},{"shards":"0,2","objects":2089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11613","reported_seq":11940,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.594923+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:20.594923+0000","last_peered":"2024-04-22T08:32:20.594923+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:20.594923+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34062336,"num_objects":8316,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2104},{"shards":"2","objects":4655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11666","reported_seq":19343,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:20.623718+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:20.623718+0000","last_peered":"2024-04-22T08:32:20.623718+0000","last_clean":"2024-04-22T08:32:20.623718+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:20.623718+0000","last_undegraded":"2024-04-22T08:32:20.623718+0000","last_fullsized":"2024-04-22T08:32:20.623718+0000","mapping_epoch":38,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33866962,"num_objects":8270,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11714","reported_seq":12108,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.457823+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:20.457823+0000","last_peered":"2024-04-22T08:32:20.457823+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:20.457823+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34521705,"num_objects":8429,"num_object_clones":0,"num_object_copies":16858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4864},{"shards":"0,2","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11600","reported_seq":12008,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.440741+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:20.440741+0000","last_peered":"2024-04-22T08:32:20.440741+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:20.440741+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1500","ondisk_log_start":"28'1500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33382400,"num_objects":8150,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4428},{"shards":"0,2","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11692","reported_seq":11983,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.463317+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:20.463317+0000","last_peered":"2024-04-22T08:32:20.463317+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:20.463317+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1600","ondisk_log_start":"28'1600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34186450,"num_objects":8348,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4749},{"shards":"0,2","objects":2034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11711","reported_seq":12144,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:20.438079+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:20.438079+0000","last_peered":"2024-04-22T08:32:20.438079+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:20.438079+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33962404,"num_objects":8295,"num_object_clones":0,"num_object_copies":16590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4557},{"shards":"0,2","objects":2177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":544096109,"num_objects":132848,"num_object_clones":0,"num_object_copies":265696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34527,"num_objects_misplaced":2169,"num_objects_unfound":0,"num_objects_dirty":132848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160856,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8911,"num_bytes_recovered":31343209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1188872192,"data_stored":1188797947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117060,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685916,"kb_used_data":387664,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93685924,"statfs":{"total":96636764160,"available":95934386176,"internally_reserved":0,"allocated":396967936,"data_stored":396742009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,2,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.7830000000000004},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.40999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920822,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617668,"kb_used_data":315324,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93754172,"statfs":{"total":96636764160,"available":96004272128,"internally_reserved":0,"allocated":322891776,"data_stored":322615418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.88}]}]},{"osd":0,"up_from":21,"seq":90194313288,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825464,"kb_used_data":460840,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93546376,"statfs":{"total":96636764160,"available":95791489024,"internally_reserved":0,"allocated":471900160,"data_stored":471684767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373354195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,9,3,0,1,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.8680000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":4.4809999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":470532096,"data_stored":470528617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322502656,"data_stored":322439441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395837440,"data_stored":395829889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:26.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:26.965 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:26.967 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:27.024 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-04-22T08:32:26.271162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117968,"kb_used_data":1152748,"kb_used_omap":12,"kb_used_meta":965171,"kb_avail":280997552,"statfs":{"total":289910292480,"available":287741493248,"internally_reserved":0,"allocated":1180413952,"data_stored":1179691041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":988335214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,8,4,8,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16740352,"num_objects":-4087,"num_object_clones":0,"num_object_copies":-8174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2317,"num_objects_misplaced":1931,"num_objects_unfound":0,"num_objects_dirty":-4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.153194"},"pg_stats":[{"pgid":"2.f","version":"39'11928","reported_seq":12316,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.364623+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:25.364623+0000","last_peered":"2024-04-22T08:32:25.364623+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:25.364623+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4607},{"shards":"0,2","objects":1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'11974","reported_seq":12092,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.592811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:25.592811+0000","last_peered":"2024-04-22T08:32:25.592811+0000","last_clean":"2024-04-22T08:32:25.592811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:25.592811+0000","last_undegraded":"2024-04-22T08:32:25.592811+0000","last_fullsized":"2024-04-22T08:32:25.592811+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33824768,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11753","reported_seq":12058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.634966+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:25.634966+0000","last_peered":"2024-04-22T08:32:25.634966+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:25.634966+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33218584,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8580,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1936},{"shards":"2","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'11931","reported_seq":11964,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.646024+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:25.646024+0000","last_peered":"2024-04-22T08:32:25.646024+0000","last_clean":"2024-04-22T08:32:25.646024+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:25.646024+0000","last_undegraded":"2024-04-22T08:32:25.646024+0000","last_fullsized":"2024-04-22T08:32:25.646024+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33517568,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11798","reported_seq":11939,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.526655+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:25.526655+0000","last_peered":"2024-04-22T08:32:25.526655+0000","last_clean":"2024-04-22T08:32:25.526655+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:25.526655+0000","last_undegraded":"2024-04-22T08:32:25.526655+0000","last_fullsized":"2024-04-22T08:32:25.526655+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33263616,"num_objects":8121,"num_object_clones":0,"num_object_copies":16242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'11879","reported_seq":12701,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.516405+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:25.516405+0000","last_peered":"2024-04-22T08:32:25.516405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:25.516405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33456128,"num_objects":8168,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1528,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":8168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4615},{"shards":"0,2","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11713","reported_seq":19451,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:24.374078+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:24.374078+0000","last_peered":"2024-04-22T08:32:24.374078+0000","last_clean":"2024-04-22T08:32:24.374078+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:24.374078+0000","last_undegraded":"2024-04-22T08:32:24.374078+0000","last_fullsized":"2024-04-22T08:32:24.374078+0000","mapping_epoch":36,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33386496,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11711","reported_seq":11858,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.432147+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:25.432147+0000","last_peered":"2024-04-22T08:32:25.432147+0000","last_clean":"2024-04-22T08:32:25.432147+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:25.432147+0000","last_undegraded":"2024-04-22T08:32:25.432147+0000","last_fullsized":"2024-04-22T08:32:25.432147+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33157120,"num_objects":8095,"num_object_clones":0,"num_object_copies":16190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12030","reported_seq":12292,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.593085+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:25.593085+0000","last_peered":"2024-04-22T08:32:25.593085+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:25.593085+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551570,"num_objects":8193,"num_object_clones":0,"num_object_copies":16386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2001},{"shards":"2","objects":4373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'11921","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.513688+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:25.513688+0000","last_peered":"2024-04-22T08:32:25.513688+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:25.513688+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854674,"num_objects":8267,"num_object_clones":0,"num_object_copies":16534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4675},{"shards":"0,2","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11745","reported_seq":12086,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.535639+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:25.535639+0000","last_peered":"2024-04-22T08:32:25.535639+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:25.535639+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33521664,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1972},{"shards":"2","objects":4530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11785","reported_seq":19462,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.570557+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:25.570557+0000","last_peered":"2024-04-22T08:32:25.570557+0000","last_clean":"2024-04-22T08:32:25.570557+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:25.570557+0000","last_undegraded":"2024-04-22T08:32:25.570557+0000","last_fullsized":"2024-04-22T08:32:25.570557+0000","mapping_epoch":38,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379538,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11864","reported_seq":12270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.423260+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:25.423260+0000","last_peered":"2024-04-22T08:32:25.423260+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:25.423260+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33907305,"num_objects":8279,"num_object_clones":0,"num_object_copies":16558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4720},{"shards":"0,2","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11732","reported_seq":12154,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.429440+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:25.429440+0000","last_peered":"2024-04-22T08:32:25.429440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:25.429440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4303},{"shards":"0,2","objects":1984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11818","reported_seq":12127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.468137+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:25.468137+0000","last_peered":"2024-04-22T08:32:25.468137+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:25.468137+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33670354,"num_objects":8222,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4632},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11833","reported_seq":12280,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.465268+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:25.465268+0000","last_peered":"2024-04-22T08:32:25.465268+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:25.465268+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33462692,"num_objects":8173,"num_object_clones":0,"num_object_copies":16346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4442},{"shards":"0,2","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177460736,"data_stored":1177386491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117061,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686012,"kb_used_data":383660,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93685828,"statfs":{"total":96636764160,"available":95934287872,"internally_reserved":0,"allocated":392867840,"data_stored":392641913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.859},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.593}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.7830000000000004},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.40999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920823,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620776,"kb_used_data":314336,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93751064,"statfs":{"total":96636764160,"available":96001089536,"internally_reserved":0,"allocated":321880064,"data_stored":321598473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":313781964},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.313},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313289,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811180,"kb_used_data":454752,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93560660,"statfs":{"total":96636764160,"available":95806115840,"internally_reserved":0,"allocated":465666048,"data_stored":465450655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,2,3,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.311},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":464195584,"data_stored":464192105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321482752,"data_stored":321419537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391782400,"data_stored":391774849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:27.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:27.745 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:27.746 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:27.808 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-04-22T08:32:26.271162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117968,"kb_used_data":1152748,"kb_used_omap":12,"kb_used_meta":965171,"kb_avail":280997552,"statfs":{"total":289910292480,"available":287741493248,"internally_reserved":0,"allocated":1180413952,"data_stored":1179691041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":988335214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,8,4,8,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16740352,"num_objects":-4087,"num_object_clones":0,"num_object_copies":-8174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2317,"num_objects_misplaced":1931,"num_objects_unfound":0,"num_objects_dirty":-4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.153194"},"pg_stats":[{"pgid":"2.f","version":"39'11928","reported_seq":12316,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.364623+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:25.364623+0000","last_peered":"2024-04-22T08:32:25.364623+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:25.364623+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4607},{"shards":"0,2","objects":1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'11974","reported_seq":12092,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.592811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:25.592811+0000","last_peered":"2024-04-22T08:32:25.592811+0000","last_clean":"2024-04-22T08:32:25.592811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:25.592811+0000","last_undegraded":"2024-04-22T08:32:25.592811+0000","last_fullsized":"2024-04-22T08:32:25.592811+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33824768,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11753","reported_seq":12058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.634966+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:25.634966+0000","last_peered":"2024-04-22T08:32:25.634966+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:25.634966+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33218584,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8580,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1936},{"shards":"2","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'11931","reported_seq":11964,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.646024+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:25.646024+0000","last_peered":"2024-04-22T08:32:25.646024+0000","last_clean":"2024-04-22T08:32:25.646024+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:25.646024+0000","last_undegraded":"2024-04-22T08:32:25.646024+0000","last_fullsized":"2024-04-22T08:32:25.646024+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33517568,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11798","reported_seq":11939,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.526655+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:25.526655+0000","last_peered":"2024-04-22T08:32:25.526655+0000","last_clean":"2024-04-22T08:32:25.526655+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:25.526655+0000","last_undegraded":"2024-04-22T08:32:25.526655+0000","last_fullsized":"2024-04-22T08:32:25.526655+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33263616,"num_objects":8121,"num_object_clones":0,"num_object_copies":16242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'11879","reported_seq":12701,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.516405+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:25.516405+0000","last_peered":"2024-04-22T08:32:25.516405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:25.516405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33456128,"num_objects":8168,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1528,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":8168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4615},{"shards":"0,2","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11713","reported_seq":19451,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:24.374078+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:24.374078+0000","last_peered":"2024-04-22T08:32:24.374078+0000","last_clean":"2024-04-22T08:32:24.374078+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:24.374078+0000","last_undegraded":"2024-04-22T08:32:24.374078+0000","last_fullsized":"2024-04-22T08:32:24.374078+0000","mapping_epoch":36,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33386496,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11711","reported_seq":11858,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.432147+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:25.432147+0000","last_peered":"2024-04-22T08:32:25.432147+0000","last_clean":"2024-04-22T08:32:25.432147+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:25.432147+0000","last_undegraded":"2024-04-22T08:32:25.432147+0000","last_fullsized":"2024-04-22T08:32:25.432147+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33157120,"num_objects":8095,"num_object_clones":0,"num_object_copies":16190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12030","reported_seq":12292,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.593085+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:25.593085+0000","last_peered":"2024-04-22T08:32:25.593085+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:25.593085+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551570,"num_objects":8193,"num_object_clones":0,"num_object_copies":16386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2001},{"shards":"2","objects":4373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'11921","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.513688+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:25.513688+0000","last_peered":"2024-04-22T08:32:25.513688+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:25.513688+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854674,"num_objects":8267,"num_object_clones":0,"num_object_copies":16534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4675},{"shards":"0,2","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11745","reported_seq":12086,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.535639+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:25.535639+0000","last_peered":"2024-04-22T08:32:25.535639+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:25.535639+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33521664,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1972},{"shards":"2","objects":4530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11785","reported_seq":19462,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.570557+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:25.570557+0000","last_peered":"2024-04-22T08:32:25.570557+0000","last_clean":"2024-04-22T08:32:25.570557+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:25.570557+0000","last_undegraded":"2024-04-22T08:32:25.570557+0000","last_fullsized":"2024-04-22T08:32:25.570557+0000","mapping_epoch":38,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379538,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11864","reported_seq":12270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.423260+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:25.423260+0000","last_peered":"2024-04-22T08:32:25.423260+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:25.423260+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33907305,"num_objects":8279,"num_object_clones":0,"num_object_copies":16558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4720},{"shards":"0,2","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11732","reported_seq":12154,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.429440+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:25.429440+0000","last_peered":"2024-04-22T08:32:25.429440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:25.429440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4303},{"shards":"0,2","objects":1984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11818","reported_seq":12127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.468137+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:25.468137+0000","last_peered":"2024-04-22T08:32:25.468137+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:25.468137+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33670354,"num_objects":8222,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4632},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11833","reported_seq":12280,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.465268+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:25.465268+0000","last_peered":"2024-04-22T08:32:25.465268+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:25.465268+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33462692,"num_objects":8173,"num_object_clones":0,"num_object_copies":16346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4442},{"shards":"0,2","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177460736,"data_stored":1177386491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117061,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686012,"kb_used_data":383660,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93685828,"statfs":{"total":96636764160,"available":95934287872,"internally_reserved":0,"allocated":392867840,"data_stored":392641913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.859},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.593}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.7830000000000004},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.40999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920823,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620776,"kb_used_data":314336,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93751064,"statfs":{"total":96636764160,"available":96001089536,"internally_reserved":0,"allocated":321880064,"data_stored":321598473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":313781964},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.313},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313289,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811180,"kb_used_data":454752,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93560660,"statfs":{"total":96636764160,"available":95806115840,"internally_reserved":0,"allocated":465666048,"data_stored":465450655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,2,3,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.311},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":464195584,"data_stored":464192105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321482752,"data_stored":321419537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391782400,"data_stored":391774849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'11928", 'reported_seq': 12316, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.364623+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:25.364623+0000', 'last_peered': '2024-04-22T08:32:25.364623+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:25.364623+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33751040, 'num_objects': 8240, 'num_object_clones': 0, 'num_object_copies': 16480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3633, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11928, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 161, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4607}, {'shards': '0,2', 'objects': 1950}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'11753", 'reported_seq': 12058, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.634966+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:25.634966+0000', 'last_peered': '2024-04-22T08:32:25.634966+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:25.634966+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33218584, 'num_objects': 8111, 'num_object_clones': 0, 'num_object_copies': 16222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3667, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8111, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8580, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 89, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1936}, {'shards': '2', 'objects': 4445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:27.810 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'11879", 'reported_seq': 12701, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.516405+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:25.516405+0000', 'last_peered': '2024-04-22T08:32:25.516405+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:25.516405+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33456128, 'num_objects': 8168, 'num_object_clones': 0, 'num_object_copies': 16336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1528, 'num_objects_misplaced': 2025, 'num_objects_unfound': 0, 'num_objects_dirty': 8168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8748, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 329, 'num_bytes_recovered': 630784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4615}, {'shards': '0,2', 'objects': 2025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12030", 'reported_seq': 12292, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.593085+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:25.593085+0000', 'last_peered': '2024-04-22T08:32:25.593085+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:25.593085+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33551570, 'num_objects': 8193, 'num_object_clones': 0, 'num_object_copies': 16386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3822, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11970, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2001}, {'shards': '2', 'objects': 4373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'11921", 'reported_seq': 12359, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.513688+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:25.513688+0000', 'last_peered': '2024-04-22T08:32:25.513688+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:25.513688+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33854674, 'num_objects': 8267, 'num_object_clones': 0, 'num_object_copies': 16534, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3594, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11919, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 182, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4675}, {'shards': '0,2', 'objects': 1947}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'11745", 'reported_seq': 12086, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.535639+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:25.535639+0000', 'last_peered': '2024-04-22T08:32:25.535639+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:25.535639+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33521664, 'num_objects': 8184, 'num_object_clones': 0, 'num_object_copies': 16368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3656, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8199, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 96, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1972}, {'shards': '2', 'objects': 4530}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:27.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'11864", 'reported_seq': 12270, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.423260+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:25.423260+0000', 'last_peered': '2024-04-22T08:32:25.423260+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:25.423260+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33907305, 'num_objects': 8279, 'num_object_clones': 0, 'num_object_copies': 16558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3560, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11863, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 164, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4720}, {'shards': '0,2', 'objects': 1929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'11732", 'reported_seq': 12154, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.429440+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:25.429440+0000', 'last_peered': '2024-04-22T08:32:25.429440+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:25.429440+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32841728, 'num_objects': 8018, 'num_object_clones': 0, 'num_object_copies': 16036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3718, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7963, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4303}, {'shards': '0,2', 'objects': 1984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'11818", 'reported_seq': 12127, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.468137+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:25.468137+0000', 'last_peered': '2024-04-22T08:32:25.468137+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:25.468137+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33670354, 'num_objects': 8222, 'num_object_clones': 0, 'num_object_copies': 16444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3592, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11816, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 117, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4632}, {'shards': '0,2', 'objects': 1908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:27.812 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'11833", 'reported_seq': 12280, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:25.465268+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:25.465268+0000', 'last_peered': '2024-04-22T08:32:25.465268+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:25.465268+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33462692, 'num_objects': 8173, 'num_object_clones': 0, 'num_object_copies': 16346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3734, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11830, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 155, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4442}, {'shards': '0,2', 'objects': 2055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:27.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:28.662 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:28.662 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34504/261628 objects degraded (13.188%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":130814,"data_bytes":535764845,"bytes_used":2168799232,"bytes_avail":287741493248,"bytes_total":289910292480,"degraded_objects":34504,"degraded_total":261628,"degraded_ratio":0.13188190866421026,"misplaced_objects":2025,"misplaced_total":261628,"misplaced_ratio":0.0077399972480009785,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":101111,"recovering_keys_per_sec":0,"num_objects_recovered":386,"num_bytes_recovered":1011712,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":408},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:28.700 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:28.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:29.337 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:29.339 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:29.386 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-04-22T08:32:28.271763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117968,"kb_used_data":1152748,"kb_used_omap":12,"kb_used_meta":965171,"kb_avail":280997552,"statfs":{"total":289910292480,"available":287741493248,"internally_reserved":0,"allocated":1180413952,"data_stored":1179691041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":988335214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,8,4,8,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16740352,"num_objects":-4087,"num_object_clones":0,"num_object_copies":-8174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2317,"num_objects_misplaced":1931,"num_objects_unfound":0,"num_objects_dirty":-4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005868"},"pg_stats":[{"pgid":"2.f","version":"39'11928","reported_seq":12316,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.364623+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:25.364623+0000","last_peered":"2024-04-22T08:32:25.364623+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:25.364623+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4607},{"shards":"0,2","objects":1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'11974","reported_seq":12092,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.592811+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:25.592811+0000","last_peered":"2024-04-22T08:32:25.592811+0000","last_clean":"2024-04-22T08:32:25.592811+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:25.592811+0000","last_undegraded":"2024-04-22T08:32:25.592811+0000","last_fullsized":"2024-04-22T08:32:25.592811+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33824768,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11753","reported_seq":12058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.634966+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:25.634966+0000","last_peered":"2024-04-22T08:32:25.634966+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:25.634966+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33218584,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8580,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1936},{"shards":"2","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'11931","reported_seq":11964,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.646024+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:25.646024+0000","last_peered":"2024-04-22T08:32:25.646024+0000","last_clean":"2024-04-22T08:32:25.646024+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:25.646024+0000","last_undegraded":"2024-04-22T08:32:25.646024+0000","last_fullsized":"2024-04-22T08:32:25.646024+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33517568,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11798","reported_seq":11939,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.526655+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:25.526655+0000","last_peered":"2024-04-22T08:32:25.526655+0000","last_clean":"2024-04-22T08:32:25.526655+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:25.526655+0000","last_undegraded":"2024-04-22T08:32:25.526655+0000","last_fullsized":"2024-04-22T08:32:25.526655+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33263616,"num_objects":8121,"num_object_clones":0,"num_object_copies":16242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'11879","reported_seq":12701,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.516405+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:25.516405+0000","last_peered":"2024-04-22T08:32:25.516405+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:25.516405+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33456128,"num_objects":8168,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1528,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":8168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4615},{"shards":"0,2","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11713","reported_seq":19451,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:24.374078+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:24.374078+0000","last_peered":"2024-04-22T08:32:24.374078+0000","last_clean":"2024-04-22T08:32:24.374078+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:24.374078+0000","last_undegraded":"2024-04-22T08:32:24.374078+0000","last_fullsized":"2024-04-22T08:32:24.374078+0000","mapping_epoch":36,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33386496,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11711","reported_seq":11858,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.432147+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:25.432147+0000","last_peered":"2024-04-22T08:32:25.432147+0000","last_clean":"2024-04-22T08:32:25.432147+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:25.432147+0000","last_undegraded":"2024-04-22T08:32:25.432147+0000","last_fullsized":"2024-04-22T08:32:25.432147+0000","mapping_epoch":27,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33157120,"num_objects":8095,"num_object_clones":0,"num_object_copies":16190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12030","reported_seq":12292,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.593085+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:25.593085+0000","last_peered":"2024-04-22T08:32:25.593085+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:25.593085+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551570,"num_objects":8193,"num_object_clones":0,"num_object_copies":16386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2001},{"shards":"2","objects":4373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'11921","reported_seq":12359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.513688+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:25.513688+0000","last_peered":"2024-04-22T08:32:25.513688+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:25.513688+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854674,"num_objects":8267,"num_object_clones":0,"num_object_copies":16534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4675},{"shards":"0,2","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11745","reported_seq":12086,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.535639+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:25.535639+0000","last_peered":"2024-04-22T08:32:25.535639+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:25.535639+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33521664,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1972},{"shards":"2","objects":4530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11785","reported_seq":19462,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:25.570557+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:25.570557+0000","last_peered":"2024-04-22T08:32:25.570557+0000","last_clean":"2024-04-22T08:32:25.570557+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:25.570557+0000","last_undegraded":"2024-04-22T08:32:25.570557+0000","last_fullsized":"2024-04-22T08:32:25.570557+0000","mapping_epoch":38,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379538,"num_objects":8151,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11864","reported_seq":12270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.423260+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:25.423260+0000","last_peered":"2024-04-22T08:32:25.423260+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:25.423260+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33907305,"num_objects":8279,"num_object_clones":0,"num_object_copies":16558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4720},{"shards":"0,2","objects":1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11732","reported_seq":12154,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.429440+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:25.429440+0000","last_peered":"2024-04-22T08:32:25.429440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:25.429440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1700","ondisk_log_start":"28'1700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4303},{"shards":"0,2","objects":1984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11818","reported_seq":12127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.468137+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:25.468137+0000","last_peered":"2024-04-22T08:32:25.468137+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:25.468137+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33670354,"num_objects":8222,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4632},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11833","reported_seq":12280,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:25.465268+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:25.465268+0000","last_peered":"2024-04-22T08:32:25.465268+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:25.465268+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33462692,"num_objects":8173,"num_object_clones":0,"num_object_copies":16346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4442},{"shards":"0,2","objects":2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":535764845,"num_objects":130814,"num_object_clones":0,"num_object_copies":261628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34504,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":130814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":31707753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177460736,"data_stored":1177386491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117061,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686012,"kb_used_data":383660,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93685828,"statfs":{"total":96636764160,"available":95934287872,"internally_reserved":0,"allocated":392867840,"data_stored":392641913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.859},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.593}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.7830000000000004},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.40999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920823,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620776,"kb_used_data":314336,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93751064,"statfs":{"total":96636764160,"available":96001089536,"internally_reserved":0,"allocated":321880064,"data_stored":321598473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":313781964},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.313},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313289,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811180,"kb_used_data":454752,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93560660,"statfs":{"total":96636764160,"available":95806115840,"internally_reserved":0,"allocated":465666048,"data_stored":465450655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,2,3,0,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.311},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":464195584,"data_stored":464192105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321482752,"data_stored":321419537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391782400,"data_stored":391774849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:32.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:33.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:33.114 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-04-22T08:32:32.276103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2123436,"kb_used_data":1141840,"kb_used_omap":12,"kb_used_meta":981555,"kb_avail":280992084,"statfs":{"total":289910292480,"available":287735894016,"internally_reserved":0,"allocated":1169244160,"data_stored":1168521249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1005112430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,5,2,0,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24682496,"num_objects":-6026,"num_object_clones":0,"num_object_copies":-12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2384,"num_objects_misplaced":1795,"num_objects_unfound":0,"num_objects_dirty":-6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":1421312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009752"},"pg_stats":[{"pgid":"2.f","version":"39'12052","reported_seq":12480,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.376620+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:30.376620+0000","last_peered":"2024-04-22T08:32:30.376620+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:30.376620+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33243136,"num_objects":8116,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4503},{"shards":"0,2","objects":1826}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12094","reported_seq":12212,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.535628+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:30.535628+0000","last_peered":"2024-04-22T08:32:30.535628+0000","last_clean":"2024-04-22T08:32:30.535628+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:30.535628+0000","last_undegraded":"2024-04-22T08:32:30.535628+0000","last_fullsized":"2024-04-22T08:32:30.535628+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33333248,"num_objects":8138,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11869","reported_seq":12190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.578028+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:30.578028+0000","last_peered":"2024-04-22T08:32:30.578028+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:30.578028+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743448,"num_objects":7995,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8696,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1820},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12065","reported_seq":12098,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.508482+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:30.508482+0000","last_peered":"2024-04-22T08:32:30.508482+0000","last_clean":"2024-04-22T08:32:30.508482+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:30.508482+0000","last_undegraded":"2024-04-22T08:32:30.508482+0000","last_fullsized":"2024-04-22T08:32:30.508482+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32968704,"num_objects":8049,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11930","reported_seq":12070,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.494885+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:30.494885+0000","last_peered":"2024-04-22T08:32:30.494885+0000","last_clean":"2024-04-22T08:32:30.494885+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:30.494885+0000","last_undegraded":"2024-04-22T08:32:30.494885+0000","last_fullsized":"2024-04-22T08:32:30.494885+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12016","reported_seq":13045,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526597+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:30.526597+0000","last_peered":"2024-04-22T08:32:30.526597+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:30.526597+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32894976,"num_objects":8031,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1559,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4583},{"shards":"0,2","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11844","reported_seq":19583,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:29.421563+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:29.421563+0000","last_peered":"2024-04-22T08:32:29.421563+0000","last_clean":"2024-04-22T08:32:29.421563+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:29.421563+0000","last_undegraded":"2024-04-22T08:32:29.421563+0000","last_fullsized":"2024-04-22T08:32:29.421563+0000","mapping_epoch":36,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32849920,"num_objects":8020,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8212,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11842","reported_seq":11989,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.434218+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:30.434218+0000","last_peered":"2024-04-22T08:32:30.434218+0000","last_clean":"2024-04-22T08:32:30.434218+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:30.434218+0000","last_undegraded":"2024-04-22T08:32:30.434218+0000","last_fullsized":"2024-04-22T08:32:30.434218+0000","mapping_epoch":27,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12158","reported_seq":12434,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.575866+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:30.575866+0000","last_peered":"2024-04-22T08:32:30.575866+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:30.575866+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027282,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1873},{"shards":"2","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12041","reported_seq":12508,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.494232+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:30.494232+0000","last_peered":"2024-04-22T08:32:30.494232+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:30.494232+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363154,"num_objects":8147,"num_object_clones":0,"num_object_copies":16294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4571},{"shards":"0,2","objects":1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11877","reported_seq":12230,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.454568+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:30.454568+0000","last_peered":"2024-04-22T08:32:30.454568+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:30.454568+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32980992,"num_objects":8052,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1840},{"shards":"2","objects":4404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11914","reported_seq":19591,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.629938+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:30.629938+0000","last_peered":"2024-04-22T08:32:30.629938+0000","last_clean":"2024-04-22T08:32:30.629938+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:30.629938+0000","last_undegraded":"2024-04-22T08:32:30.629938+0000","last_fullsized":"2024-04-22T08:32:30.629938+0000","mapping_epoch":38,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32851154,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12001","reported_seq":12441,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.490951+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:30.490951+0000","last_peered":"2024-04-22T08:32:30.490951+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:30.490951+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33346153,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11839","reported_seq":12271,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.479658+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:30.479658+0000","last_peered":"2024-04-22T08:32:30.479658+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:30.479658+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32403456,"num_objects":7911,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4201},{"shards":"0,2","objects":1877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11955","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526030+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:30.526030+0000","last_peered":"2024-04-22T08:32:30.526030+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:30.526030+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33109202,"num_objects":8085,"num_object_clones":0,"num_object_copies":16170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11977","reported_seq":12452,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.458910+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:30.458910+0000","last_peered":"2024-04-22T08:32:30.458910+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:30.458910+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32872868,"num_objects":8029,"num_object_clones":0,"num_object_copies":16058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11974,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4312},{"shards":"0,2","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165950976,"data_stored":1165876731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117062,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":379668,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":388780032,"data_stored":388554105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.2450000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.2730000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.4430000000000001},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.177}]}]},{"osd":1,"up_from":33,"seq":141733920824,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628056,"kb_used_data":313424,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93743784,"statfs":{"total":96636764160,"available":95993634816,"internally_reserved":0,"allocated":320946176,"data_stored":320664585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":322170572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809272,"kb_used_data":448748,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93562568,"statfs":{"total":96636764160,"available":95808069632,"internally_reserved":0,"allocated":459517952,"data_stored":459302559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.1909999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.314},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457871360,"data_stored":457867881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320487424,"data_stored":320424209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387592192,"data_stored":387584641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:33.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:33.902 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:33.943 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-04-22T08:32:32.276103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2123436,"kb_used_data":1141840,"kb_used_omap":12,"kb_used_meta":981555,"kb_avail":280992084,"statfs":{"total":289910292480,"available":287735894016,"internally_reserved":0,"allocated":1169244160,"data_stored":1168521249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1005112430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,5,2,0,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24682496,"num_objects":-6026,"num_object_clones":0,"num_object_copies":-12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2384,"num_objects_misplaced":1795,"num_objects_unfound":0,"num_objects_dirty":-6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":1421312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009752"},"pg_stats":[{"pgid":"2.f","version":"39'12052","reported_seq":12480,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.376620+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:30.376620+0000","last_peered":"2024-04-22T08:32:30.376620+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:30.376620+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33243136,"num_objects":8116,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4503},{"shards":"0,2","objects":1826}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12094","reported_seq":12212,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.535628+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:30.535628+0000","last_peered":"2024-04-22T08:32:30.535628+0000","last_clean":"2024-04-22T08:32:30.535628+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:30.535628+0000","last_undegraded":"2024-04-22T08:32:30.535628+0000","last_fullsized":"2024-04-22T08:32:30.535628+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33333248,"num_objects":8138,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11869","reported_seq":12190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.578028+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:30.578028+0000","last_peered":"2024-04-22T08:32:30.578028+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:30.578028+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743448,"num_objects":7995,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8696,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1820},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12065","reported_seq":12098,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.508482+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:30.508482+0000","last_peered":"2024-04-22T08:32:30.508482+0000","last_clean":"2024-04-22T08:32:30.508482+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:30.508482+0000","last_undegraded":"2024-04-22T08:32:30.508482+0000","last_fullsized":"2024-04-22T08:32:30.508482+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32968704,"num_objects":8049,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11930","reported_seq":12070,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.494885+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:30.494885+0000","last_peered":"2024-04-22T08:32:30.494885+0000","last_clean":"2024-04-22T08:32:30.494885+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:30.494885+0000","last_undegraded":"2024-04-22T08:32:30.494885+0000","last_fullsized":"2024-04-22T08:32:30.494885+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12016","reported_seq":13045,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526597+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:30.526597+0000","last_peered":"2024-04-22T08:32:30.526597+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:30.526597+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32894976,"num_objects":8031,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1559,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4583},{"shards":"0,2","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11844","reported_seq":19583,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:29.421563+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:29.421563+0000","last_peered":"2024-04-22T08:32:29.421563+0000","last_clean":"2024-04-22T08:32:29.421563+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:29.421563+0000","last_undegraded":"2024-04-22T08:32:29.421563+0000","last_fullsized":"2024-04-22T08:32:29.421563+0000","mapping_epoch":36,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32849920,"num_objects":8020,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8212,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11842","reported_seq":11989,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.434218+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:30.434218+0000","last_peered":"2024-04-22T08:32:30.434218+0000","last_clean":"2024-04-22T08:32:30.434218+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:30.434218+0000","last_undegraded":"2024-04-22T08:32:30.434218+0000","last_fullsized":"2024-04-22T08:32:30.434218+0000","mapping_epoch":27,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12158","reported_seq":12434,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.575866+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:30.575866+0000","last_peered":"2024-04-22T08:32:30.575866+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:30.575866+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027282,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1873},{"shards":"2","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12041","reported_seq":12508,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.494232+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:30.494232+0000","last_peered":"2024-04-22T08:32:30.494232+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:30.494232+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363154,"num_objects":8147,"num_object_clones":0,"num_object_copies":16294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4571},{"shards":"0,2","objects":1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11877","reported_seq":12230,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.454568+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:30.454568+0000","last_peered":"2024-04-22T08:32:30.454568+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:30.454568+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32980992,"num_objects":8052,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1840},{"shards":"2","objects":4404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11914","reported_seq":19591,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.629938+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:30.629938+0000","last_peered":"2024-04-22T08:32:30.629938+0000","last_clean":"2024-04-22T08:32:30.629938+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:30.629938+0000","last_undegraded":"2024-04-22T08:32:30.629938+0000","last_fullsized":"2024-04-22T08:32:30.629938+0000","mapping_epoch":38,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32851154,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12001","reported_seq":12441,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.490951+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:30.490951+0000","last_peered":"2024-04-22T08:32:30.490951+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:30.490951+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33346153,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11839","reported_seq":12271,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.479658+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:30.479658+0000","last_peered":"2024-04-22T08:32:30.479658+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:30.479658+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32403456,"num_objects":7911,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4201},{"shards":"0,2","objects":1877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11955","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526030+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:30.526030+0000","last_peered":"2024-04-22T08:32:30.526030+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:30.526030+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33109202,"num_objects":8085,"num_object_clones":0,"num_object_copies":16170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11977","reported_seq":12452,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.458910+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:30.458910+0000","last_peered":"2024-04-22T08:32:30.458910+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:30.458910+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32872868,"num_objects":8029,"num_object_clones":0,"num_object_copies":16058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11974,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4312},{"shards":"0,2","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165950976,"data_stored":1165876731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117062,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":379668,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":388780032,"data_stored":388554105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.2450000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.2730000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.4430000000000001},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.177}]}]},{"osd":1,"up_from":33,"seq":141733920824,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628056,"kb_used_data":313424,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93743784,"statfs":{"total":96636764160,"available":95993634816,"internally_reserved":0,"allocated":320946176,"data_stored":320664585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":322170572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809272,"kb_used_data":448748,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93562568,"statfs":{"total":96636764160,"available":95808069632,"internally_reserved":0,"allocated":459517952,"data_stored":459302559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.1909999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.314},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457871360,"data_stored":457867881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320487424,"data_stored":320424209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387592192,"data_stored":387584641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12052", 'reported_seq': 12480, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.376620+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:30.376620+0000', 'last_peered': '2024-04-22T08:32:30.376620+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:30.376620+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33243136, 'num_objects': 8116, 'num_object_clones': 0, 'num_object_copies': 16232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3613, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12052, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 181, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4503}, {'shards': '0,2', 'objects': 1826}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'11869", 'reported_seq': 12190, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.578028+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:30.578028+0000', 'last_peered': '2024-04-22T08:32:30.578028+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:30.578028+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32743448, 'num_objects': 7995, 'num_object_clones': 0, 'num_object_copies': 15990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3659, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7995, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8696, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1820}, {'shards': '2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:33.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12016", 'reported_seq': 13045, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.526597+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:30.526597+0000', 'last_peered': '2024-04-22T08:32:30.526597+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:30.526597+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32894976, 'num_objects': 8031, 'num_object_clones': 0, 'num_object_copies': 16062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1559, 'num_objects_misplaced': 1889, 'num_objects_unfound': 0, 'num_objects_dirty': 8031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8885, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 434, 'num_bytes_recovered': 1040384, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4583}, {'shards': '0,2', 'objects': 1889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12158", 'reported_seq': 12434, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.575866+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:30.575866+0000', 'last_peered': '2024-04-22T08:32:30.575866+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:30.575866+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33027282, 'num_objects': 8065, 'num_object_clones': 0, 'num_object_copies': 16130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3815, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1873}, {'shards': '2', 'objects': 4252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12041", 'reported_seq': 12508, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.494232+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:30.494232+0000', 'last_peered': '2024-04-22T08:32:30.494232+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:30.494232+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33363154, 'num_objects': 8147, 'num_object_clones': 0, 'num_object_copies': 16294, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3578, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12039, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 198, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4571}, {'shards': '0,2', 'objects': 1827}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'11877", 'reported_seq': 12230, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.454568+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:30.454568+0000', 'last_peered': '2024-04-22T08:32:30.454568+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:30.454568+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32980992, 'num_objects': 8052, 'num_object_clones': 0, 'num_object_copies': 16104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3650, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8331, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1840}, {'shards': '2', 'objects': 4404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12001", 'reported_seq': 12441, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.490951+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:30.490951+0000', 'last_peered': '2024-04-22T08:32:30.490951+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:30.490951+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33346153, 'num_objects': 8142, 'num_object_clones': 0, 'num_object_copies': 16284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3543, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12000, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 181, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4600}, {'shards': '0,2', 'objects': 1792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:33.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'11839", 'reported_seq': 12271, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.479658+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:30.479658+0000', 'last_peered': '2024-04-22T08:32:30.479658+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:30.479658+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32403456, 'num_objects': 7911, 'num_object_clones': 0, 'num_object_copies': 15822, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3713, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8070, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4201}, {'shards': '0,2', 'objects': 1877}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:33.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'11955", 'reported_seq': 12274, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.526030+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:30.526030+0000', 'last_peered': '2024-04-22T08:32:30.526030+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:30.526030+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33109202, 'num_objects': 8085, 'num_object_clones': 0, 'num_object_copies': 16170, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3587, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11953, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4500}, {'shards': '0,2', 'objects': 1771}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:33.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'11977", 'reported_seq': 12452, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:30.458910+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:30.458910+0000', 'last_peered': '2024-04-22T08:32:30.458910+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:30.458910+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32872868, 'num_objects': 8029, 'num_object_clones': 0, 'num_object_copies': 16058, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11974, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 169, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4312}, {'shards': '0,2', 'objects': 1911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:33.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:34.712 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34437/257510 objects degraded (13.373%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":389,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":128755,"data_bytes":527331181,"bytes_used":2174398464,"bytes_avail":287735894016,"bytes_total":289910292480,"degraded_objects":34437,"degraded_total":257510,"degraded_ratio":0.13373072890373189,"misplaced_objects":1889,"misplaced_total":257510,"misplaced_ratio":0.0073356374509727781,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":64459,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":774144,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":340},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:34.804 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:34.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:35.434 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:35.434 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:35.484 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-04-22T08:32:34.276510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2123436,"kb_used_data":1141840,"kb_used_omap":12,"kb_used_meta":981555,"kb_avail":280992084,"statfs":{"total":289910292480,"available":287735894016,"internally_reserved":0,"allocated":1169244160,"data_stored":1168521249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1005112430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,5,2,0,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16764928,"num_objects":-4093,"num_object_clones":0,"num_object_copies":-8186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":-280,"num_objects_unfound":0,"num_objects_dirty":-4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":774144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009686"},"pg_stats":[{"pgid":"2.f","version":"39'12052","reported_seq":12480,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.376620+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:30.376620+0000","last_peered":"2024-04-22T08:32:30.376620+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:30.376620+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33243136,"num_objects":8116,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4503},{"shards":"0,2","objects":1826}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12094","reported_seq":12212,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.535628+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:30.535628+0000","last_peered":"2024-04-22T08:32:30.535628+0000","last_clean":"2024-04-22T08:32:30.535628+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:30.535628+0000","last_undegraded":"2024-04-22T08:32:30.535628+0000","last_fullsized":"2024-04-22T08:32:30.535628+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33333248,"num_objects":8138,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11869","reported_seq":12190,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.578028+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:30.578028+0000","last_peered":"2024-04-22T08:32:30.578028+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:30.578028+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743448,"num_objects":7995,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8696,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1820},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12065","reported_seq":12098,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.508482+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:30.508482+0000","last_peered":"2024-04-22T08:32:30.508482+0000","last_clean":"2024-04-22T08:32:30.508482+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:30.508482+0000","last_undegraded":"2024-04-22T08:32:30.508482+0000","last_fullsized":"2024-04-22T08:32:30.508482+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32968704,"num_objects":8049,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'11930","reported_seq":12070,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.494885+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:30.494885+0000","last_peered":"2024-04-22T08:32:30.494885+0000","last_clean":"2024-04-22T08:32:30.494885+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:30.494885+0000","last_undegraded":"2024-04-22T08:32:30.494885+0000","last_fullsized":"2024-04-22T08:32:30.494885+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12016","reported_seq":13045,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526597+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:30.526597+0000","last_peered":"2024-04-22T08:32:30.526597+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:30.526597+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32894976,"num_objects":8031,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1559,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4583},{"shards":"0,2","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11844","reported_seq":19583,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:29.421563+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:29.421563+0000","last_peered":"2024-04-22T08:32:29.421563+0000","last_clean":"2024-04-22T08:32:29.421563+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:29.421563+0000","last_undegraded":"2024-04-22T08:32:29.421563+0000","last_fullsized":"2024-04-22T08:32:29.421563+0000","mapping_epoch":36,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32849920,"num_objects":8020,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8212,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11842","reported_seq":11989,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.434218+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:30.434218+0000","last_peered":"2024-04-22T08:32:30.434218+0000","last_clean":"2024-04-22T08:32:30.434218+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:30.434218+0000","last_undegraded":"2024-04-22T08:32:30.434218+0000","last_fullsized":"2024-04-22T08:32:30.434218+0000","mapping_epoch":27,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12158","reported_seq":12434,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.575866+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:30.575866+0000","last_peered":"2024-04-22T08:32:30.575866+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:30.575866+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027282,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1873},{"shards":"2","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12041","reported_seq":12508,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.494232+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:30.494232+0000","last_peered":"2024-04-22T08:32:30.494232+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:30.494232+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363154,"num_objects":8147,"num_object_clones":0,"num_object_copies":16294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4571},{"shards":"0,2","objects":1827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'11877","reported_seq":12230,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.454568+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:30.454568+0000","last_peered":"2024-04-22T08:32:30.454568+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:30.454568+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32980992,"num_objects":8052,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1840},{"shards":"2","objects":4404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'11914","reported_seq":19591,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:30.629938+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:30.629938+0000","last_peered":"2024-04-22T08:32:30.629938+0000","last_clean":"2024-04-22T08:32:30.629938+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:30.629938+0000","last_undegraded":"2024-04-22T08:32:30.629938+0000","last_fullsized":"2024-04-22T08:32:30.629938+0000","mapping_epoch":38,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32851154,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12001","reported_seq":12441,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.490951+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:30.490951+0000","last_peered":"2024-04-22T08:32:30.490951+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:30.490951+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33346153,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11839","reported_seq":12271,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.479658+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:30.479658+0000","last_peered":"2024-04-22T08:32:30.479658+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:30.479658+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1800","ondisk_log_start":"28'1800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32403456,"num_objects":7911,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4201},{"shards":"0,2","objects":1877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'11955","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.526030+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:30.526030+0000","last_peered":"2024-04-22T08:32:30.526030+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:30.526030+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33109202,"num_objects":8085,"num_object_clones":0,"num_object_copies":16170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'11977","reported_seq":12452,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:30.458910+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:30.458910+0000","last_peered":"2024-04-22T08:32:30.458910+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:30.458910+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32872868,"num_objects":8029,"num_object_clones":0,"num_object_copies":16058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11974,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4312},{"shards":"0,2","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":527331181,"num_objects":128755,"num_object_clones":0,"num_object_copies":257510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34437,"num_objects_misplaced":1889,"num_objects_unfound":0,"num_objects_dirty":128755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164949,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9281,"num_bytes_recovered":32117353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165950976,"data_stored":1165876731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117062,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":379668,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":388780032,"data_stored":388554105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":3.2450000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":3.2730000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.4430000000000001},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":2.177}]}]},{"osd":1,"up_from":33,"seq":141733920824,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628056,"kb_used_data":313424,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93743784,"statfs":{"total":96636764160,"available":95993634816,"internally_reserved":0,"allocated":320946176,"data_stored":320664585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":322170572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809272,"kb_used_data":448748,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93562568,"statfs":{"total":96636764160,"available":95808069632,"internally_reserved":0,"allocated":459517952,"data_stored":459302559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.1909999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.314},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457871360,"data_stored":457867881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320487424,"data_stored":320424209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387592192,"data_stored":387584641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:38.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:39.097 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:39.104 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:39.141 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-04-22T08:32:38.277578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519081837,"num_objects":126741,"num_object_clones":0,"num_object_copies":253482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166963,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2107060,"kb_used_data":1129564,"kb_used_omap":12,"kb_used_meta":977459,"kb_avail":281008460,"statfs":{"total":289910292480,"available":287752663040,"internally_reserved":0,"allocated":1156673536,"data_stored":1155950625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1000918126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,0,6,4,5,0,10,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16683008,"num_objects":-4073,"num_object_clones":0,"num_object_copies":-8146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":-284,"num_objects_unfound":0,"num_objects_dirty":-4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":806912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006416"},"pg_stats":[{"pgid":"2.f","version":"39'12168","reported_seq":12620,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.431574+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:35.431574+0000","last_peered":"2024-04-22T08:32:35.431574+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:35.431574+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4399},{"shards":"0,2","objects":1710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12211","reported_seq":12329,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.645582+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:35.645582+0000","last_peered":"2024-04-22T08:32:35.645582+0000","last_clean":"2024-04-22T08:32:35.645582+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:35.645582+0000","last_undegraded":"2024-04-22T08:32:35.645582+0000","last_fullsized":"2024-04-22T08:32:35.645582+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32854016,"num_objects":8021,"num_object_clones":0,"num_object_copies":16042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11982","reported_seq":12315,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.529868+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:35.529868+0000","last_peered":"2024-04-22T08:32:35.529868+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:35.529868+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280600,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8809,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1707},{"shards":"2","objects":4230}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12192","reported_seq":12225,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.598427+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:35.598427+0000","last_peered":"2024-04-22T08:32:35.598427+0000","last_clean":"2024-04-22T08:32:35.598427+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:35.598427+0000","last_undegraded":"2024-04-22T08:32:35.598427+0000","last_fullsized":"2024-04-22T08:32:35.598427+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12045","reported_seq":12186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.395883+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:35.395883+0000","last_peered":"2024-04-22T08:32:35.395883+0000","last_clean":"2024-04-22T08:32:35.395883+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:35.395883+0000","last_undegraded":"2024-04-22T08:32:35.395883+0000","last_fullsized":"2024-04-22T08:32:35.395883+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":7874,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12161","reported_seq":13413,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.534563+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:35.534563+0000","last_peered":"2024-04-22T08:32:35.534563+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:35.534563+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1597,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":1437696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4548},{"shards":"0,2","objects":1741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11959","reported_seq":19698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:34.401413+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:34.401413+0000","last_peered":"2024-04-22T08:32:34.401413+0000","last_clean":"2024-04-22T08:32:34.401413+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:34.401413+0000","last_undegraded":"2024-04-22T08:32:34.401413+0000","last_fullsized":"2024-04-22T08:32:34.401413+0000","mapping_epoch":36,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32378880,"num_objects":7905,"num_object_clones":0,"num_object_copies":15810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11986","reported_seq":12133,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.479693+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:35.479693+0000","last_peered":"2024-04-22T08:32:35.479693+0000","last_clean":"2024-04-22T08:32:35.479693+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:35.479693+0000","last_undegraded":"2024-04-22T08:32:35.479693+0000","last_fullsized":"2024-04-22T08:32:35.479693+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32030720,"num_objects":7820,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12294","reported_seq":12584,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.497191+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:35.497191+0000","last_peered":"2024-04-22T08:32:35.497191+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:35.497191+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470226,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1737},{"shards":"2","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12164","reported_seq":12688,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.523376+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:35.523376+0000","last_peered":"2024-04-22T08:32:35.523376+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:35.523376+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32859346,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4475},{"shards":"0,2","objects":1704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12000","reported_seq":12357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.560451+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:35.560451+0000","last_peered":"2024-04-22T08:32:35.560451+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:35.560451+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1717},{"shards":"2","objects":4283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12040","reported_seq":19717,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.517911+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:35.517911+0000","last_peered":"2024-04-22T08:32:35.517911+0000","last_clean":"2024-04-22T08:32:35.517911+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:35.517911+0000","last_undegraded":"2024-04-22T08:32:35.517911+0000","last_fullsized":"2024-04-22T08:32:35.517911+0000","mapping_epoch":38,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32335058,"num_objects":7896,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12113","reported_seq":12577,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.450245+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:35.450245+0000","last_peered":"2024-04-22T08:32:35.450245+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:35.450245+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32887401,"num_objects":8030,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12112,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11969","reported_seq":12419,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.550644+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:35.550644+0000","last_peered":"2024-04-22T08:32:35.550644+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:35.550644+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31870976,"num_objects":7781,"num_object_clones":0,"num_object_copies":15562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4080},{"shards":"0,2","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12084","reported_seq":12443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.415626+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:35.415626+0000","last_peered":"2024-04-22T08:32:35.415626+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:35.415626+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32580818,"num_objects":7956,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4391},{"shards":"0,2","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12120","reported_seq":12629,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.494861+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:35.494861+0000","last_peered":"2024-04-22T08:32:35.494861+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:35.494861+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32287140,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4186},{"shards":"0,2","objects":1768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":519081837,"num_objects":126741,"num_object_clones":0,"num_object_copies":253482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166963,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1154482176,"data_stored":1154407931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117063,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685380,"kb_used_data":374844,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93686460,"statfs":{"total":96636764160,"available":95934935040,"internally_reserved":0,"allocated":383840256,"data_stored":383614329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,1,3,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.22}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":18.32}]}]},{"osd":1,"up_from":33,"seq":141733920825,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614632,"kb_used_data":312288,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93757208,"statfs":{"total":96636764160,"available":96007380992,"internally_reserved":0,"allocated":319782912,"data_stored":319501321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.188},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313291,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807048,"kb_used_data":442432,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93564792,"statfs":{"total":96636764160,"available":95810347008,"internally_reserved":0,"allocated":453050368,"data_stored":452834975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373354195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,4,0,0,0,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":6.3620000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":451637248,"data_stored":451633769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319356928,"data_stored":319293713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383488000,"data_stored":383480449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:39.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:39.851 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:39.865 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:39.942 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-04-22T08:32:38.277578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519081837,"num_objects":126741,"num_object_clones":0,"num_object_copies":253482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166963,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2107060,"kb_used_data":1129564,"kb_used_omap":12,"kb_used_meta":977459,"kb_avail":281008460,"statfs":{"total":289910292480,"available":287752663040,"internally_reserved":0,"allocated":1156673536,"data_stored":1155950625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1000918126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,0,6,4,5,0,10,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16683008,"num_objects":-4073,"num_object_clones":0,"num_object_copies":-8146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":-284,"num_objects_unfound":0,"num_objects_dirty":-4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":806912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006416"},"pg_stats":[{"pgid":"2.f","version":"39'12168","reported_seq":12620,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.431574+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:35.431574+0000","last_peered":"2024-04-22T08:32:35.431574+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:35.431574+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4399},{"shards":"0,2","objects":1710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12211","reported_seq":12329,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.645582+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:35.645582+0000","last_peered":"2024-04-22T08:32:35.645582+0000","last_clean":"2024-04-22T08:32:35.645582+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:35.645582+0000","last_undegraded":"2024-04-22T08:32:35.645582+0000","last_fullsized":"2024-04-22T08:32:35.645582+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32854016,"num_objects":8021,"num_object_clones":0,"num_object_copies":16042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11982","reported_seq":12315,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.529868+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:35.529868+0000","last_peered":"2024-04-22T08:32:35.529868+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:35.529868+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280600,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8809,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1707},{"shards":"2","objects":4230}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12192","reported_seq":12225,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.598427+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:35.598427+0000","last_peered":"2024-04-22T08:32:35.598427+0000","last_clean":"2024-04-22T08:32:35.598427+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:35.598427+0000","last_undegraded":"2024-04-22T08:32:35.598427+0000","last_fullsized":"2024-04-22T08:32:35.598427+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12045","reported_seq":12186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.395883+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:35.395883+0000","last_peered":"2024-04-22T08:32:35.395883+0000","last_clean":"2024-04-22T08:32:35.395883+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:35.395883+0000","last_undegraded":"2024-04-22T08:32:35.395883+0000","last_fullsized":"2024-04-22T08:32:35.395883+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":7874,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12161","reported_seq":13413,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.534563+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:35.534563+0000","last_peered":"2024-04-22T08:32:35.534563+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:35.534563+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1597,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":1437696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4548},{"shards":"0,2","objects":1741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'11959","reported_seq":19698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:34.401413+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:34.401413+0000","last_peered":"2024-04-22T08:32:34.401413+0000","last_clean":"2024-04-22T08:32:34.401413+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:34.401413+0000","last_undegraded":"2024-04-22T08:32:34.401413+0000","last_fullsized":"2024-04-22T08:32:34.401413+0000","mapping_epoch":36,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32378880,"num_objects":7905,"num_object_clones":0,"num_object_copies":15810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11986","reported_seq":12133,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.479693+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:35.479693+0000","last_peered":"2024-04-22T08:32:35.479693+0000","last_clean":"2024-04-22T08:32:35.479693+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:35.479693+0000","last_undegraded":"2024-04-22T08:32:35.479693+0000","last_fullsized":"2024-04-22T08:32:35.479693+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32030720,"num_objects":7820,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12294","reported_seq":12584,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.497191+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:35.497191+0000","last_peered":"2024-04-22T08:32:35.497191+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:35.497191+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470226,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1737},{"shards":"2","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12164","reported_seq":12688,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.523376+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:35.523376+0000","last_peered":"2024-04-22T08:32:35.523376+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:35.523376+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32859346,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4475},{"shards":"0,2","objects":1704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12000","reported_seq":12357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.560451+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:35.560451+0000","last_peered":"2024-04-22T08:32:35.560451+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:35.560451+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1717},{"shards":"2","objects":4283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12040","reported_seq":19717,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.517911+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:35.517911+0000","last_peered":"2024-04-22T08:32:35.517911+0000","last_clean":"2024-04-22T08:32:35.517911+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:35.517911+0000","last_undegraded":"2024-04-22T08:32:35.517911+0000","last_fullsized":"2024-04-22T08:32:35.517911+0000","mapping_epoch":38,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32335058,"num_objects":7896,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12113","reported_seq":12577,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.450245+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:35.450245+0000","last_peered":"2024-04-22T08:32:35.450245+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:35.450245+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32887401,"num_objects":8030,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12112,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11969","reported_seq":12419,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.550644+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:35.550644+0000","last_peered":"2024-04-22T08:32:35.550644+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:35.550644+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31870976,"num_objects":7781,"num_object_clones":0,"num_object_copies":15562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4080},{"shards":"0,2","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12084","reported_seq":12443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.415626+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:35.415626+0000","last_peered":"2024-04-22T08:32:35.415626+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:35.415626+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32580818,"num_objects":7956,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4391},{"shards":"0,2","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12120","reported_seq":12629,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.494861+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:35.494861+0000","last_peered":"2024-04-22T08:32:35.494861+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:35.494861+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32287140,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4186},{"shards":"0,2","objects":1768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":519081837,"num_objects":126741,"num_object_clones":0,"num_object_copies":253482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166963,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1154482176,"data_stored":1154407931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117063,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685380,"kb_used_data":374844,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93686460,"statfs":{"total":96636764160,"available":95934935040,"internally_reserved":0,"allocated":383840256,"data_stored":383614329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,1,3,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.22}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":18.32}]}]},{"osd":1,"up_from":33,"seq":141733920825,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614632,"kb_used_data":312288,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93757208,"statfs":{"total":96636764160,"available":96007380992,"internally_reserved":0,"allocated":319782912,"data_stored":319501321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":309587660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.188},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313291,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807048,"kb_used_data":442432,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93564792,"statfs":{"total":96636764160,"available":95810347008,"internally_reserved":0,"allocated":453050368,"data_stored":452834975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373354195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,4,0,0,0,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":6.3620000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":451637248,"data_stored":451633769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319356928,"data_stored":319293713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383488000,"data_stored":383480449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12168", 'reported_seq': 12620, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.431574+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:35.431574+0000', 'last_peered': '2024-04-22T08:32:35.431574+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:35.431574+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32768000, 'num_objects': 8000, 'num_object_clones': 0, 'num_object_copies': 16000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 193, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4399}, {'shards': '0,2', 'objects': 1710}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'11982", 'reported_seq': 12315, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.529868+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:35.529868+0000', 'last_peered': '2024-04-22T08:32:35.529868+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:35.529868+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32280600, 'num_objects': 7882, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3653, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7882, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8809, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1707}, {'shards': '2', 'objects': 4230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12161", 'reported_seq': 13413, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.534563+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:35.534563+0000', 'last_peered': '2024-04-22T08:32:35.534563+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:35.534563+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32301056, 'num_objects': 7886, 'num_object_clones': 0, 'num_object_copies': 15772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1597, 'num_objects_misplaced': 1741, 'num_objects_unfound': 0, 'num_objects_dirty': 7886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9030, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 544, 'num_bytes_recovered': 1437696, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4548}, {'shards': '0,2', 'objects': 1741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:39.944 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12294", 'reported_seq': 12584, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.497191+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:35.497191+0000', 'last_peered': '2024-04-22T08:32:35.497191+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:35.497191+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32470226, 'num_objects': 7929, 'num_object_clones': 0, 'num_object_copies': 15858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3808, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12234, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1737}, {'shards': '2', 'objects': 4123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12164", 'reported_seq': 12688, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.523376+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:35.523376+0000', 'last_peered': '2024-04-22T08:32:35.523376+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:35.523376+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32859346, 'num_objects': 8024, 'num_object_clones': 0, 'num_object_copies': 16048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3551, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12162, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 225, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4475}, {'shards': '0,2', 'objects': 1704}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12000", 'reported_seq': 12357, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.560451+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:35.560451+0000', 'last_peered': '2024-04-22T08:32:35.560451+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:35.560451+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32477184, 'num_objects': 7929, 'num_object_clones': 0, 'num_object_copies': 15858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3648, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8454, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 104, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1717}, {'shards': '2', 'objects': 4283}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12113", 'reported_seq': 12577, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.450245+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:35.450245+0000', 'last_peered': '2024-04-22T08:32:35.450245+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:35.450245+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32887401, 'num_objects': 8030, 'num_object_clones': 0, 'num_object_copies': 16060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3531, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12112, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 193, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4500}, {'shards': '0,2', 'objects': 1680}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'11969", 'reported_seq': 12419, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.550644+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:35.550644+0000', 'last_peered': '2024-04-22T08:32:35.550644+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:35.550644+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31870976, 'num_objects': 7781, 'num_object_clones': 0, 'num_object_copies': 15562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3704, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8200, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 136, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4080}, {'shards': '0,2', 'objects': 1747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:39.945 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12084", 'reported_seq': 12443, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.415626+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:35.415626+0000', 'last_peered': '2024-04-22T08:32:35.415626+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:35.415626+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32580818, 'num_objects': 7956, 'num_object_clones': 0, 'num_object_copies': 15912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3567, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12082, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 142, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4391}, {'shards': '0,2', 'objects': 1642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:39.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12120", 'reported_seq': 12629, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:35.494861+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:35.494861+0000', 'last_peered': '2024-04-22T08:32:35.494861+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:35.494861+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32287140, 'num_objects': 7886, 'num_object_clones': 0, 'num_object_copies': 15772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3703, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12117, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 186, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4186}, {'shards': '0,2', 'objects': 1768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:39.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:40.683 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:40.683 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34363/253242 objects degraded (13.569%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":395,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":126621,"data_bytes":518590317,"bytes_used":2183524352,"bytes_avail":287726768128,"bytes_total":289910292480,"degraded_objects":34363,"degraded_total":253242,"degraded_ratio":0.13569234171267008,"misplaced_objects":1741,"misplaced_total":253242,"misplaced_ratio":0.0068748469843075014,"recovering_objects_per_sec":35,"recovering_bytes_per_sec":67207,"recovering_keys_per_sec":0,"num_objects_recovered":425,"num_bytes_recovered":806912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":349},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:40.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:40.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:41.452 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:41.452 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:41.514 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-04-22T08:32:40.278053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":518590317,"num_objects":126621,"num_object_clones":0,"num_object_copies":253242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167083,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2132348,"kb_used_data":1128612,"kb_used_omap":12,"kb_used_meta":1003699,"kb_avail":280983172,"statfs":{"total":289910292480,"available":287726768128,"internally_reserved":0,"allocated":1155698688,"data_stored":1154975777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1027787886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,0,5,1,3,0,10,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17174528,"num_objects":-4193,"num_object_clones":0,"num_object_copies":-8386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":-284,"num_objects_unfound":0,"num_objects_dirty":-4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":806912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006290"},"pg_stats":[{"pgid":"2.f","version":"39'12168","reported_seq":12620,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.431574+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:35.431574+0000","last_peered":"2024-04-22T08:32:35.431574+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:35.431574+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4399},{"shards":"0,2","objects":1710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12211","reported_seq":12329,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.645582+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:35.645582+0000","last_peered":"2024-04-22T08:32:35.645582+0000","last_clean":"2024-04-22T08:32:35.645582+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:35.645582+0000","last_undegraded":"2024-04-22T08:32:35.645582+0000","last_fullsized":"2024-04-22T08:32:35.645582+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32854016,"num_objects":8021,"num_object_clones":0,"num_object_copies":16042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'11982","reported_seq":12315,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.529868+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:35.529868+0000","last_peered":"2024-04-22T08:32:35.529868+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:35.529868+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280600,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8809,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1707},{"shards":"2","objects":4230}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12192","reported_seq":12225,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.598427+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:35.598427+0000","last_peered":"2024-04-22T08:32:35.598427+0000","last_clean":"2024-04-22T08:32:35.598427+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:35.598427+0000","last_undegraded":"2024-04-22T08:32:35.598427+0000","last_fullsized":"2024-04-22T08:32:35.598427+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12045","reported_seq":12186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.395883+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:35.395883+0000","last_peered":"2024-04-22T08:32:35.395883+0000","last_clean":"2024-04-22T08:32:35.395883+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:35.395883+0000","last_undegraded":"2024-04-22T08:32:35.395883+0000","last_fullsized":"2024-04-22T08:32:35.395883+0000","mapping_epoch":27,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":7874,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12161","reported_seq":13413,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.534563+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:35.534563+0000","last_peered":"2024-04-22T08:32:35.534563+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:35.534563+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1597,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":1437696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4548},{"shards":"0,2","objects":1741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12079","reported_seq":19818,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:39.351346+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:39.351346+0000","last_peered":"2024-04-22T08:32:39.351346+0000","last_clean":"2024-04-22T08:32:39.351346+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:39.351346+0000","last_undegraded":"2024-04-22T08:32:39.351346+0000","last_fullsized":"2024-04-22T08:32:39.351346+0000","mapping_epoch":36,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'11986","reported_seq":12133,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.479693+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:35.479693+0000","last_peered":"2024-04-22T08:32:35.479693+0000","last_clean":"2024-04-22T08:32:35.479693+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:35.479693+0000","last_undegraded":"2024-04-22T08:32:35.479693+0000","last_fullsized":"2024-04-22T08:32:35.479693+0000","mapping_epoch":27,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32030720,"num_objects":7820,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12294","reported_seq":12584,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.497191+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:35.497191+0000","last_peered":"2024-04-22T08:32:35.497191+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:35.497191+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470226,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1737},{"shards":"2","objects":4123}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12164","reported_seq":12688,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.523376+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:35.523376+0000","last_peered":"2024-04-22T08:32:35.523376+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:35.523376+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32859346,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4475},{"shards":"0,2","objects":1704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12000","reported_seq":12357,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.560451+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:35.560451+0000","last_peered":"2024-04-22T08:32:35.560451+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:35.560451+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1717},{"shards":"2","objects":4283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12040","reported_seq":19717,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:35.517911+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:35.517911+0000","last_peered":"2024-04-22T08:32:35.517911+0000","last_clean":"2024-04-22T08:32:35.517911+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:35.517911+0000","last_undegraded":"2024-04-22T08:32:35.517911+0000","last_fullsized":"2024-04-22T08:32:35.517911+0000","mapping_epoch":38,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32335058,"num_objects":7896,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12113","reported_seq":12577,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.450245+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:35.450245+0000","last_peered":"2024-04-22T08:32:35.450245+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:35.450245+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32887401,"num_objects":8030,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12112,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4500},{"shards":"0,2","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'11969","reported_seq":12419,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.550644+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:35.550644+0000","last_peered":"2024-04-22T08:32:35.550644+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:35.550644+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'1900","ondisk_log_start":"28'1900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31870976,"num_objects":7781,"num_object_clones":0,"num_object_copies":15562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4080},{"shards":"0,2","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12084","reported_seq":12443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.415626+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:35.415626+0000","last_peered":"2024-04-22T08:32:35.415626+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:35.415626+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32580818,"num_objects":7956,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4391},{"shards":"0,2","objects":1642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12120","reported_seq":12629,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:35.494861+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:35.494861+0000","last_peered":"2024-04-22T08:32:35.494861+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:35.494861+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32287140,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4186},{"shards":"0,2","objects":1768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":518590317,"num_objects":126621,"num_object_clones":0,"num_object_copies":253242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34363,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":126621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167083,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9503,"num_bytes_recovered":32514665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1153511424,"data_stored":1153437179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117063,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685380,"kb_used_data":374844,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93686460,"statfs":{"total":96636764160,"available":95934935040,"internally_reserved":0,"allocated":383840256,"data_stored":383614329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,1,3,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.22}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":18.32}]}]},{"osd":1,"up_from":33,"seq":141733920826,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639920,"kb_used_data":311336,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93731920,"statfs":{"total":96636764160,"available":95981486080,"internally_reserved":0,"allocated":318808064,"data_stored":318526473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.439},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313291,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807048,"kb_used_data":442432,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93564792,"statfs":{"total":96636764160,"available":95810347008,"internally_reserved":0,"allocated":453050368,"data_stored":452834975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":373354195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,4,0,0,0,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":6.3620000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":451637248,"data_stored":451633769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318386176,"data_stored":318322961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383488000,"data_stored":383480449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:44.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:45.142 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:45.145 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:45.191 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-22T08:32:44.283616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":510783341,"num_objects":124715,"num_object_clones":0,"num_object_copies":249430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34315,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":124715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168989,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9699,"num_bytes_recovered":32911977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2101560,"kb_used_data":1118288,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281013960,"statfs":{"total":289910292480,"available":287758295040,"internally_reserved":0,"allocated":1145126912,"data_stored":1144404001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,7,1,7,1,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16547840,"num_objects":-4040,"num_object_clones":0,"num_object_copies":-8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":-296,"num_objects_unfound":0,"num_objects_dirty":-4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007513"},"pg_stats":[{"pgid":"2.f","version":"39'12287","reported_seq":12762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.531482+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:40.531482+0000","last_peered":"2024-04-22T08:32:40.531482+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:40.531482+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280576,"num_objects":7881,"num_object_clones":0,"num_object_copies":15762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4292},{"shards":"0,2","objects":1591}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12351","reported_seq":12469,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.544902+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:40.544902+0000","last_peered":"2024-04-22T08:32:40.544902+0000","last_clean":"2024-04-22T08:32:40.544902+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:40.544902+0000","last_undegraded":"2024-04-22T08:32:40.544902+0000","last_fullsized":"2024-04-22T08:32:40.544902+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280576,"num_objects":7881,"num_object_clones":0,"num_object_copies":15762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12107","reported_seq":12442,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.611722+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:40.611722+0000","last_peered":"2024-04-22T08:32:40.611722+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:40.611722+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31768600,"num_objects":7757,"num_object_clones":0,"num_object_copies":15514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7757,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8934,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1582},{"shards":"2","objects":4106}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12316","reported_seq":12349,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.638240+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:40.638240+0000","last_peered":"2024-04-22T08:32:40.638240+0000","last_clean":"2024-04-22T08:32:40.638240+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:40.638240+0000","last_undegraded":"2024-04-22T08:32:40.638240+0000","last_fullsized":"2024-04-22T08:32:40.638240+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":7798,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12180","reported_seq":12321,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.463972+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:40.463972+0000","last_peered":"2024-04-22T08:32:40.463972+0000","last_clean":"2024-04-22T08:32:40.463972+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:40.463972+0000","last_undegraded":"2024-04-22T08:32:40.463972+0000","last_fullsized":"2024-04-22T08:32:40.463972+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31698944,"num_objects":7739,"num_object_clones":0,"num_object_copies":15478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12308","reported_seq":13766,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.585028+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:40.585028+0000","last_peered":"2024-04-22T08:32:40.585028+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:40.585028+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31698944,"num_objects":7739,"num_object_clones":0,"num_object_copies":15478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1642,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":7739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12079","reported_seq":19818,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:39.351346+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:39.351346+0000","last_peered":"2024-04-22T08:32:39.351346+0000","last_clean":"2024-04-22T08:32:39.351346+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:39.351346+0000","last_undegraded":"2024-04-22T08:32:39.351346+0000","last_fullsized":"2024-04-22T08:32:39.351346+0000","mapping_epoch":36,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12119","reported_seq":12266,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.496205+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:40.496205+0000","last_peered":"2024-04-22T08:32:40.496205+0000","last_clean":"2024-04-22T08:32:40.496205+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:40.496205+0000","last_undegraded":"2024-04-22T08:32:40.496205+0000","last_fullsized":"2024-04-22T08:32:40.496205+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31485952,"num_objects":7687,"num_object_clones":0,"num_object_copies":15374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12417","reported_seq":12704,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.665691+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:40.665691+0000","last_peered":"2024-04-22T08:32:40.665691+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:40.665691+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31966418,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1614},{"shards":"2","objects":4000}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12274","reported_seq":12840,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.572771+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:40.572771+0000","last_peered":"2024-04-22T08:32:40.572771+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:40.572771+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32408786,"num_objects":7914,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4386},{"shards":"0,2","objects":1594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12112","reported_seq":12483,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.621013+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:40.621013+0000","last_peered":"2024-04-22T08:32:40.621013+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:40.621013+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32018432,"num_objects":7817,"num_object_clones":0,"num_object_copies":15634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8566,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1605},{"shards":"2","objects":4178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12175","reported_seq":19852,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.590133+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:40.590133+0000","last_peered":"2024-04-22T08:32:40.590133+0000","last_clean":"2024-04-22T08:32:40.590133+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:40.590133+0000","last_undegraded":"2024-04-22T08:32:40.590133+0000","last_fullsized":"2024-04-22T08:32:40.590133+0000","mapping_epoch":38,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782098,"num_objects":7761,"num_object_clones":0,"num_object_copies":15522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12229","reported_seq":12715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.532798+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:40.532798+0000","last_peered":"2024-04-22T08:32:40.532798+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:40.532798+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32412265,"num_objects":7914,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4395},{"shards":"0,2","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12094","reported_seq":12558,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.599785+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:40.599785+0000","last_peered":"2024-04-22T08:32:40.599785+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:40.599785+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31358976,"num_objects":7656,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3962},{"shards":"0,2","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12202","reported_seq":12587,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.604582+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:40.604582+0000","last_peered":"2024-04-22T08:32:40.604582+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:40.604582+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32097490,"num_objects":7838,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12200,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4286},{"shards":"0,2","objects":1524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12264","reported_seq":12815,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.573256+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:40.573256+0000","last_peered":"2024-04-22T08:32:40.573256+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:40.573256+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31697316,"num_objects":7742,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4063},{"shards":"0,2","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":510783341,"num_objects":124715,"num_object_clones":0,"num_object_copies":249430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34315,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":124715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168989,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9699,"num_bytes_recovered":32911977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1143021568,"data_stored":1142947323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117064,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669020,"kb_used_data":370768,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93702820,"statfs":{"total":96636764160,"available":95951687680,"internally_reserved":0,"allocated":379666432,"data_stored":379440505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.308},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.798},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.27500000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920826,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639920,"kb_used_data":311336,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93731920,"statfs":{"total":96636764160,"available":95981486080,"internally_reserved":0,"allocated":318808064,"data_stored":318526473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.439},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313292,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792620,"kb_used_data":436184,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93579220,"statfs":{"total":96636764160,"available":95825121280,"internally_reserved":0,"allocated":446652416,"data_stored":446437023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,2,0,6,1,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":3.214},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.7469999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":3.1760000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445313024,"data_stored":445309545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318386176,"data_stored":318322961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379322368,"data_stored":379314817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:45.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:45.866 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:45.866 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:45.916 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-22T08:32:44.283616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":510783341,"num_objects":124715,"num_object_clones":0,"num_object_copies":249430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34315,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":124715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168989,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9699,"num_bytes_recovered":32911977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2101560,"kb_used_data":1118288,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281013960,"statfs":{"total":289910292480,"available":287758295040,"internally_reserved":0,"allocated":1145126912,"data_stored":1144404001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,7,1,7,1,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16547840,"num_objects":-4040,"num_object_clones":0,"num_object_copies":-8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":-296,"num_objects_unfound":0,"num_objects_dirty":-4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007513"},"pg_stats":[{"pgid":"2.f","version":"39'12287","reported_seq":12762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.531482+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:40.531482+0000","last_peered":"2024-04-22T08:32:40.531482+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:40.531482+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280576,"num_objects":7881,"num_object_clones":0,"num_object_copies":15762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4292},{"shards":"0,2","objects":1591}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12351","reported_seq":12469,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.544902+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:40.544902+0000","last_peered":"2024-04-22T08:32:40.544902+0000","last_clean":"2024-04-22T08:32:40.544902+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:40.544902+0000","last_undegraded":"2024-04-22T08:32:40.544902+0000","last_fullsized":"2024-04-22T08:32:40.544902+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32280576,"num_objects":7881,"num_object_clones":0,"num_object_copies":15762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12107","reported_seq":12442,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.611722+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:40.611722+0000","last_peered":"2024-04-22T08:32:40.611722+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:40.611722+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31768600,"num_objects":7757,"num_object_clones":0,"num_object_copies":15514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7757,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8934,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1582},{"shards":"2","objects":4106}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12316","reported_seq":12349,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.638240+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:40.638240+0000","last_peered":"2024-04-22T08:32:40.638240+0000","last_clean":"2024-04-22T08:32:40.638240+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:40.638240+0000","last_undegraded":"2024-04-22T08:32:40.638240+0000","last_fullsized":"2024-04-22T08:32:40.638240+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":7798,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12180","reported_seq":12321,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.463972+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:40.463972+0000","last_peered":"2024-04-22T08:32:40.463972+0000","last_clean":"2024-04-22T08:32:40.463972+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:40.463972+0000","last_undegraded":"2024-04-22T08:32:40.463972+0000","last_fullsized":"2024-04-22T08:32:40.463972+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31698944,"num_objects":7739,"num_object_clones":0,"num_object_copies":15478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9272,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12308","reported_seq":13766,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.585028+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:40.585028+0000","last_peered":"2024-04-22T08:32:40.585028+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:40.585028+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31698944,"num_objects":7739,"num_object_clones":0,"num_object_copies":15478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1642,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":7739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12079","reported_seq":19818,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:39.351346+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:39.351346+0000","last_peered":"2024-04-22T08:32:39.351346+0000","last_clean":"2024-04-22T08:32:39.351346+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:39.351346+0000","last_undegraded":"2024-04-22T08:32:39.351346+0000","last_fullsized":"2024-04-22T08:32:39.351346+0000","mapping_epoch":36,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12119","reported_seq":12266,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.496205+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:40.496205+0000","last_peered":"2024-04-22T08:32:40.496205+0000","last_clean":"2024-04-22T08:32:40.496205+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:40.496205+0000","last_undegraded":"2024-04-22T08:32:40.496205+0000","last_fullsized":"2024-04-22T08:32:40.496205+0000","mapping_epoch":27,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31485952,"num_objects":7687,"num_object_clones":0,"num_object_copies":15374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12417","reported_seq":12704,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.665691+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:40.665691+0000","last_peered":"2024-04-22T08:32:40.665691+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:40.665691+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31966418,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1614},{"shards":"2","objects":4000}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12274","reported_seq":12840,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.572771+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:40.572771+0000","last_peered":"2024-04-22T08:32:40.572771+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:40.572771+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32408786,"num_objects":7914,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4386},{"shards":"0,2","objects":1594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12112","reported_seq":12483,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.621013+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:40.621013+0000","last_peered":"2024-04-22T08:32:40.621013+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:40.621013+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32018432,"num_objects":7817,"num_object_clones":0,"num_object_copies":15634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8566,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1605},{"shards":"2","objects":4178}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12175","reported_seq":19852,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:40.590133+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:40.590133+0000","last_peered":"2024-04-22T08:32:40.590133+0000","last_clean":"2024-04-22T08:32:40.590133+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:40.590133+0000","last_undegraded":"2024-04-22T08:32:40.590133+0000","last_fullsized":"2024-04-22T08:32:40.590133+0000","mapping_epoch":38,"log_start":"28'2100","ondisk_log_start":"28'2100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782098,"num_objects":7761,"num_object_clones":0,"num_object_copies":15522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12229","reported_seq":12715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.532798+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:40.532798+0000","last_peered":"2024-04-22T08:32:40.532798+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:40.532798+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32412265,"num_objects":7914,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4395},{"shards":"0,2","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12094","reported_seq":12558,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.599785+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:40.599785+0000","last_peered":"2024-04-22T08:32:40.599785+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:40.599785+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2000","ondisk_log_start":"28'2000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31358976,"num_objects":7656,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3962},{"shards":"0,2","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12202","reported_seq":12587,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.604582+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:40.604582+0000","last_peered":"2024-04-22T08:32:40.604582+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:40.604582+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32097490,"num_objects":7838,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12200,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4286},{"shards":"0,2","objects":1524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12264","reported_seq":12815,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:40.573256+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:40.573256+0000","last_peered":"2024-04-22T08:32:40.573256+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:40.573256+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31697316,"num_objects":7742,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4063},{"shards":"0,2","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":510783341,"num_objects":124715,"num_object_clones":0,"num_object_copies":249430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34315,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":124715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168989,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9699,"num_bytes_recovered":32911977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1143021568,"data_stored":1142947323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117064,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669020,"kb_used_data":370768,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93702820,"statfs":{"total":96636764160,"available":95951687680,"internally_reserved":0,"allocated":379666432,"data_stored":379440505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.308},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.798},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.27500000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920826,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639920,"kb_used_data":311336,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93731920,"statfs":{"total":96636764160,"available":95981486080,"internally_reserved":0,"allocated":318808064,"data_stored":318526473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.439},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313292,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792620,"kb_used_data":436184,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93579220,"statfs":{"total":96636764160,"available":95825121280,"internally_reserved":0,"allocated":446652416,"data_stored":446437023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,2,0,6,1,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":3.214},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.7469999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":3.1760000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445313024,"data_stored":445309545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318386176,"data_stored":318322961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379322368,"data_stored":379314817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12287", 'reported_seq': 12762, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.531482+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:40.531482+0000', 'last_peered': '2024-04-22T08:32:40.531482+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:40.531482+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32280576, 'num_objects': 7881, 'num_object_clones': 0, 'num_object_copies': 15762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3589, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12287, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 205, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4292}, {'shards': '0,2', 'objects': 1591}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12107", 'reported_seq': 12442, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.611722+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:40.611722+0000', 'last_peered': '2024-04-22T08:32:40.611722+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:40.611722+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31768600, 'num_objects': 7757, 'num_object_clones': 0, 'num_object_copies': 15514, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3652, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7757, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8934, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 104, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1582}, {'shards': '2', 'objects': 4106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:45.918 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12308", 'reported_seq': 13766, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.585028+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:40.585028+0000', 'last_peered': '2024-04-22T08:32:40.585028+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:40.585028+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31698944, 'num_objects': 7739, 'num_object_clones': 0, 'num_object_copies': 15478, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1642, 'num_objects_misplaced': 1593, 'num_objects_unfound': 0, 'num_objects_dirty': 7739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9177, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 647, 'num_bytes_recovered': 1835008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4504}, {'shards': '0,2', 'objects': 1593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12417", 'reported_seq': 12704, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.665691+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:40.665691+0000', 'last_peered': '2024-04-22T08:32:40.665691+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:40.665691+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31966418, 'num_objects': 7806, 'num_object_clones': 0, 'num_object_copies': 15612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3808, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12357, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1614}, {'shards': '2', 'objects': 4000}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12274", 'reported_seq': 12840, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.572771+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:40.572771+0000', 'last_peered': '2024-04-22T08:32:40.572771+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:40.572771+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32408786, 'num_objects': 7914, 'num_object_clones': 0, 'num_object_copies': 15828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3530, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12272, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 246, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4386}, {'shards': '0,2', 'objects': 1594}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12112", 'reported_seq': 12483, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.621013+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:40.621013+0000', 'last_peered': '2024-04-22T08:32:40.621013+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:40.621013+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32018432, 'num_objects': 7817, 'num_object_clones': 0, 'num_object_copies': 15634, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3641, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8566, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1605}, {'shards': '2', 'objects': 4178}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12229", 'reported_seq': 12715, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.532798+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:40.532798+0000', 'last_peered': '2024-04-22T08:32:40.532798+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:40.532798+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32412265, 'num_objects': 7914, 'num_object_clones': 0, 'num_object_copies': 15828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3520, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12228, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 204, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4395}, {'shards': '0,2', 'objects': 1564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12094", 'reported_seq': 12558, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.599785+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:40.599785+0000', 'last_peered': '2024-04-22T08:32:40.599785+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:40.599785+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31358976, 'num_objects': 7656, 'num_object_clones': 0, 'num_object_copies': 15312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3697, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8325, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 143, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3962}, {'shards': '0,2', 'objects': 1622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.919 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:45.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12202", 'reported_seq': 12587, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.604582+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:40.604582+0000', 'last_peered': '2024-04-22T08:32:40.604582+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:40.604582+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32097490, 'num_objects': 7838, 'num_object_clones': 0, 'num_object_copies': 15676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3554, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12200, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 155, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4286}, {'shards': '0,2', 'objects': 1524}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:45.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12264", 'reported_seq': 12815, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:40.573256+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:40.573256+0000', 'last_peered': '2024-04-22T08:32:40.573256+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:40.573256+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31697316, 'num_objects': 7742, 'num_object_clones': 0, 'num_object_copies': 15484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3682, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12261, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 207, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4063}, {'shards': '0,2', 'objects': 1624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:45.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:46.689 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:46.689 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34268/245110 objects degraded (13.981%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":122555,"data_bytes":501935981,"bytes_used":2139869184,"bytes_avail":287770423296,"bytes_total":289910292480,"degraded_objects":34268,"degraded_total":245110,"degraded_ratio":0.13980661743706907,"misplaced_objects":1465,"misplaced_total":245110,"misplaced_ratio":0.0059769083268736475,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":99265,"recovering_keys_per_sec":0,"num_objects_recovered":593,"num_bytes_recovered":1191936,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":516},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:46.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:46.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:47.446 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:47.446 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:47.535 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-04-22T08:32:46.284071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":501935981,"num_objects":122555,"num_object_clones":0,"num_object_copies":245110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171149,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2089716,"kb_used_data":1106460,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281025804,"statfs":{"total":289910292480,"available":287770423296,"internally_reserved":0,"allocated":1133015040,"data_stored":1132292129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,12,9,3,0,1,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25395200,"num_objects":-6200,"num_object_clones":0,"num_object_copies":-12400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":-6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":593,"num_bytes_recovered":1191936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007560"},"pg_stats":[{"pgid":"2.f","version":"39'12420","reported_seq":12906,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.519650+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:45.519650+0000","last_peered":"2024-04-22T08:32:45.519650+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:45.519650+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12420,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4164},{"shards":"0,2","objects":1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12481","reported_seq":12599,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.532586+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:45.532586+0000","last_peered":"2024-04-22T08:32:45.532586+0000","last_clean":"2024-04-22T08:32:45.532586+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:45.532586+0000","last_undegraded":"2024-04-22T08:32:45.532586+0000","last_fullsized":"2024-04-22T08:32:45.532586+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31748096,"num_objects":7751,"num_object_clones":0,"num_object_copies":15502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12237","reported_seq":12586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.639658+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:45.639658+0000","last_peered":"2024-04-22T08:32:45.639658+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:45.639658+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31236120,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9064,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1452},{"shards":"2","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12461","reported_seq":12494,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.637595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:45.637595+0000","last_peered":"2024-04-22T08:32:45.637595+0000","last_clean":"2024-04-22T08:32:45.637595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:45.637595+0000","last_undegraded":"2024-04-22T08:32:45.637595+0000","last_fullsized":"2024-04-22T08:32:45.637595+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12312","reported_seq":12453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.576331+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:45.576331+0000","last_peered":"2024-04-22T08:32:45.576331+0000","last_clean":"2024-04-22T08:32:45.576331+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:45.576331+0000","last_undegraded":"2024-04-22T08:32:45.576331+0000","last_fullsized":"2024-04-22T08:32:45.576331+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31158272,"num_objects":7607,"num_object_clones":0,"num_object_copies":15214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12439","reported_seq":14093,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.598436+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:45.598436+0000","last_peered":"2024-04-22T08:32:45.598436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:45.598436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":7608,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1672,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":7608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2232320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4471},{"shards":"0,2","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12229","reported_seq":19968,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:44.471401+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:44.471401+0000","last_peered":"2024-04-22T08:32:44.471401+0000","last_clean":"2024-04-22T08:32:44.471401+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:44.471401+0000","last_undegraded":"2024-04-22T08:32:44.471401+0000","last_fullsized":"2024-04-22T08:32:44.471401+0000","mapping_epoch":36,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31272960,"num_objects":7635,"num_object_clones":0,"num_object_copies":15270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8597,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12253","reported_seq":12400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.620620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:45.620620+0000","last_peered":"2024-04-22T08:32:45.620620+0000","last_clean":"2024-04-22T08:32:45.620620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:45.620620+0000","last_undegraded":"2024-04-22T08:32:45.620620+0000","last_fullsized":"2024-04-22T08:32:45.620620+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30937088,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12559","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.660296+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:45.660296+0000","last_peered":"2024-04-22T08:32:45.660296+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:45.660296+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31384786,"num_objects":7664,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1472},{"shards":"2","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12417","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.562815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:45.562815+0000","last_peered":"2024-04-22T08:32:45.562815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:45.562815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31823058,"num_objects":7771,"num_object_clones":0,"num_object_copies":15542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12415,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4257},{"shards":"0,2","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12254","reported_seq":12645,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.608922+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:45.608922+0000","last_peered":"2024-04-22T08:32:45.608922+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:45.608922+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31436800,"num_objects":7675,"num_object_clones":0,"num_object_copies":15350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8708,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1463},{"shards":"2","objects":4046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12319","reported_seq":19996,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.621173+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:45.621173+0000","last_peered":"2024-04-22T08:32:45.621173+0000","last_clean":"2024-04-22T08:32:45.621173+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:45.621173+0000","last_undegraded":"2024-04-22T08:32:45.621173+0000","last_fullsized":"2024-04-22T08:32:45.621173+0000","mapping_epoch":38,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31192274,"num_objects":7617,"num_object_clones":0,"num_object_copies":15234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12317,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12360","reported_seq":12868,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.617493+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:45.617493+0000","last_peered":"2024-04-22T08:32:45.617493+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:45.617493+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31875689,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4275},{"shards":"0,2","objects":1433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12205","reported_seq":12677,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.584964+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:45.584964+0000","last_peered":"2024-04-22T08:32:45.584964+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:45.584964+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30904320,"num_objects":7545,"num_object_clones":0,"num_object_copies":15090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3855},{"shards":"0,2","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12318","reported_seq":12717,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.543543+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:45.543543+0000","last_peered":"2024-04-22T08:32:45.543543+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:45.543543+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31622354,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4177},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12410","reported_seq":12977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.513157+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:45.513157+0000","last_peered":"2024-04-22T08:32:45.513157+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:45.513157+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31099300,"num_objects":7596,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3925},{"shards":"0,2","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":501935981,"num_objects":122555,"num_object_clones":0,"num_object_copies":245110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171149,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130860544,"data_stored":1130786299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117065,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668736,"kb_used_data":366392,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93703104,"statfs":{"total":96636764160,"available":95951978496,"internally_reserved":0,"allocated":375185408,"data_stored":374959481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.253}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920827,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630600,"kb_used_data":310208,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93741240,"statfs":{"total":96636764160,"available":95991029760,"internally_reserved":0,"allocated":317652992,"data_stored":317371401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":328068812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,8,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.443},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.40100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313293,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790380,"kb_used_data":429860,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93581460,"statfs":{"total":96636764160,"available":95827415040,"internally_reserved":0,"allocated":440176640,"data_stored":439961247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.16},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.3360000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":5.3760000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":438812672,"data_stored":438809193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317235200,"data_stored":317171985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374812672,"data_stored":374805121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:50.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:51.278 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:51.280 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:51.308 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-04-22T08:32:50.285031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":501444461,"num_objects":122435,"num_object_clones":0,"num_object_copies":244870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171269,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092708,"kb_used_data":1105356,"kb_used_omap":12,"kb_used_meta":987315,"kb_avail":281022812,"statfs":{"total":289910292480,"available":287767359488,"internally_reserved":0,"allocated":1131884544,"data_stored":1131161633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1011010670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,6,7,0,0,1,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17637376,"num_objects":-4306,"num_object_clones":0,"num_object_copies":-8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":-276,"num_objects_unfound":0,"num_objects_dirty":-4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007452"},"pg_stats":[{"pgid":"2.f","version":"39'12420","reported_seq":12906,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.519650+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:45.519650+0000","last_peered":"2024-04-22T08:32:45.519650+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:45.519650+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12420,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4164},{"shards":"0,2","objects":1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12481","reported_seq":12599,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.532586+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:45.532586+0000","last_peered":"2024-04-22T08:32:45.532586+0000","last_clean":"2024-04-22T08:32:45.532586+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:45.532586+0000","last_undegraded":"2024-04-22T08:32:45.532586+0000","last_fullsized":"2024-04-22T08:32:45.532586+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31748096,"num_objects":7751,"num_object_clones":0,"num_object_copies":15502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12237","reported_seq":12586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.639658+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:45.639658+0000","last_peered":"2024-04-22T08:32:45.639658+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:45.639658+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31236120,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9064,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1452},{"shards":"2","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12461","reported_seq":12494,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.637595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:45.637595+0000","last_peered":"2024-04-22T08:32:45.637595+0000","last_clean":"2024-04-22T08:32:45.637595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:45.637595+0000","last_undegraded":"2024-04-22T08:32:45.637595+0000","last_fullsized":"2024-04-22T08:32:45.637595+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12312","reported_seq":12453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.576331+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:45.576331+0000","last_peered":"2024-04-22T08:32:45.576331+0000","last_clean":"2024-04-22T08:32:45.576331+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:45.576331+0000","last_undegraded":"2024-04-22T08:32:45.576331+0000","last_fullsized":"2024-04-22T08:32:45.576331+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31158272,"num_objects":7607,"num_object_clones":0,"num_object_copies":15214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12439","reported_seq":14093,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.598436+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:45.598436+0000","last_peered":"2024-04-22T08:32:45.598436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:45.598436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":7608,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1672,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":7608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2232320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4471},{"shards":"0,2","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12349","reported_seq":20088,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:49.333154+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:49.333154+0000","last_peered":"2024-04-22T08:32:49.333154+0000","last_clean":"2024-04-22T08:32:49.333154+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:49.333154+0000","last_undegraded":"2024-04-22T08:32:49.333154+0000","last_fullsized":"2024-04-22T08:32:49.333154+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30781440,"num_objects":7515,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8717,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12253","reported_seq":12400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.620620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:45.620620+0000","last_peered":"2024-04-22T08:32:45.620620+0000","last_clean":"2024-04-22T08:32:45.620620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:45.620620+0000","last_undegraded":"2024-04-22T08:32:45.620620+0000","last_fullsized":"2024-04-22T08:32:45.620620+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30937088,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12559","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.660296+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:45.660296+0000","last_peered":"2024-04-22T08:32:45.660296+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:45.660296+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31384786,"num_objects":7664,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1472},{"shards":"2","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12417","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.562815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:45.562815+0000","last_peered":"2024-04-22T08:32:45.562815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:45.562815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31823058,"num_objects":7771,"num_object_clones":0,"num_object_copies":15542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12415,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4257},{"shards":"0,2","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12254","reported_seq":12645,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.608922+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:45.608922+0000","last_peered":"2024-04-22T08:32:45.608922+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:45.608922+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31436800,"num_objects":7675,"num_object_clones":0,"num_object_copies":15350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8708,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1463},{"shards":"2","objects":4046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12319","reported_seq":19996,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.621173+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:45.621173+0000","last_peered":"2024-04-22T08:32:45.621173+0000","last_clean":"2024-04-22T08:32:45.621173+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:45.621173+0000","last_undegraded":"2024-04-22T08:32:45.621173+0000","last_fullsized":"2024-04-22T08:32:45.621173+0000","mapping_epoch":38,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31192274,"num_objects":7617,"num_object_clones":0,"num_object_copies":15234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12317,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12360","reported_seq":12868,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.617493+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:45.617493+0000","last_peered":"2024-04-22T08:32:45.617493+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:45.617493+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31875689,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4275},{"shards":"0,2","objects":1433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12205","reported_seq":12677,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.584964+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:45.584964+0000","last_peered":"2024-04-22T08:32:45.584964+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:45.584964+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30904320,"num_objects":7545,"num_object_clones":0,"num_object_copies":15090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3855},{"shards":"0,2","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12318","reported_seq":12717,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.543543+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:45.543543+0000","last_peered":"2024-04-22T08:32:45.543543+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:45.543543+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31622354,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4177},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12410","reported_seq":12977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.513157+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:45.513157+0000","last_peered":"2024-04-22T08:32:45.513157+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:45.513157+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31099300,"num_objects":7596,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3925},{"shards":"0,2","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":501444461,"num_objects":122435,"num_object_clones":0,"num_object_copies":244870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171269,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1129746432,"data_stored":1129672187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117065,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668736,"kb_used_data":366392,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93703104,"statfs":{"total":96636764160,"available":95951978496,"internally_reserved":0,"allocated":375185408,"data_stored":374959481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.253}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920828,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633592,"kb_used_data":309104,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93738248,"statfs":{"total":96636764160,"available":95987965952,"internally_reserved":0,"allocated":316522496,"data_stored":316240905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.2229999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.2619999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":4.2830000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313293,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790380,"kb_used_data":429860,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93581460,"statfs":{"total":96636764160,"available":95827415040,"internally_reserved":0,"allocated":440176640,"data_stored":439961247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.16},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.3360000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":5.3760000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":438812672,"data_stored":438809193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316121088,"data_stored":316057873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374812672,"data_stored":374805121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:51.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:52.077 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:52.078 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:52.150 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-04-22T08:32:50.285031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":501444461,"num_objects":122435,"num_object_clones":0,"num_object_copies":244870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171269,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092708,"kb_used_data":1105356,"kb_used_omap":12,"kb_used_meta":987315,"kb_avail":281022812,"statfs":{"total":289910292480,"available":287767359488,"internally_reserved":0,"allocated":1131884544,"data_stored":1131161633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1011010670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,6,7,0,0,1,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17637376,"num_objects":-4306,"num_object_clones":0,"num_object_copies":-8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":-276,"num_objects_unfound":0,"num_objects_dirty":-4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007452"},"pg_stats":[{"pgid":"2.f","version":"39'12420","reported_seq":12906,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.519650+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:45.519650+0000","last_peered":"2024-04-22T08:32:45.519650+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:45.519650+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12420,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4164},{"shards":"0,2","objects":1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12481","reported_seq":12599,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.532586+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:45.532586+0000","last_peered":"2024-04-22T08:32:45.532586+0000","last_clean":"2024-04-22T08:32:45.532586+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:45.532586+0000","last_undegraded":"2024-04-22T08:32:45.532586+0000","last_fullsized":"2024-04-22T08:32:45.532586+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31748096,"num_objects":7751,"num_object_clones":0,"num_object_copies":15502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12237","reported_seq":12586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.639658+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:45.639658+0000","last_peered":"2024-04-22T08:32:45.639658+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:45.639658+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31236120,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9064,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1452},{"shards":"2","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12461","reported_seq":12494,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.637595+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:45.637595+0000","last_peered":"2024-04-22T08:32:45.637595+0000","last_clean":"2024-04-22T08:32:45.637595+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:45.637595+0000","last_undegraded":"2024-04-22T08:32:45.637595+0000","last_fullsized":"2024-04-22T08:32:45.637595+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12312","reported_seq":12453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.576331+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:45.576331+0000","last_peered":"2024-04-22T08:32:45.576331+0000","last_clean":"2024-04-22T08:32:45.576331+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:45.576331+0000","last_undegraded":"2024-04-22T08:32:45.576331+0000","last_fullsized":"2024-04-22T08:32:45.576331+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31158272,"num_objects":7607,"num_object_clones":0,"num_object_copies":15214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12439","reported_seq":14093,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.598436+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:45.598436+0000","last_peered":"2024-04-22T08:32:45.598436+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:45.598436+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":7608,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1672,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":7608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":2232320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4471},{"shards":"0,2","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12349","reported_seq":20088,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:49.333154+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:49.333154+0000","last_peered":"2024-04-22T08:32:49.333154+0000","last_clean":"2024-04-22T08:32:49.333154+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:49.333154+0000","last_undegraded":"2024-04-22T08:32:49.333154+0000","last_fullsized":"2024-04-22T08:32:49.333154+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30781440,"num_objects":7515,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8717,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12253","reported_seq":12400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.620620+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:45.620620+0000","last_peered":"2024-04-22T08:32:45.620620+0000","last_clean":"2024-04-22T08:32:45.620620+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:45.620620+0000","last_undegraded":"2024-04-22T08:32:45.620620+0000","last_fullsized":"2024-04-22T08:32:45.620620+0000","mapping_epoch":27,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30937088,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12559","reported_seq":12871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.660296+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:45.660296+0000","last_peered":"2024-04-22T08:32:45.660296+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:45.660296+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31384786,"num_objects":7664,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1472},{"shards":"2","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12417","reported_seq":13011,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.562815+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:45.562815+0000","last_peered":"2024-04-22T08:32:45.562815+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:45.562815+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31823058,"num_objects":7771,"num_object_clones":0,"num_object_copies":15542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12415,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4257},{"shards":"0,2","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12254","reported_seq":12645,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.608922+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:45.608922+0000","last_peered":"2024-04-22T08:32:45.608922+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:45.608922+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31436800,"num_objects":7675,"num_object_clones":0,"num_object_copies":15350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8708,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1463},{"shards":"2","objects":4046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12319","reported_seq":19996,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:45.621173+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:45.621173+0000","last_peered":"2024-04-22T08:32:45.621173+0000","last_clean":"2024-04-22T08:32:45.621173+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:45.621173+0000","last_undegraded":"2024-04-22T08:32:45.621173+0000","last_fullsized":"2024-04-22T08:32:45.621173+0000","mapping_epoch":38,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31192274,"num_objects":7617,"num_object_clones":0,"num_object_copies":15234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12317,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12360","reported_seq":12868,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.617493+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:45.617493+0000","last_peered":"2024-04-22T08:32:45.617493+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:45.617493+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31875689,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4275},{"shards":"0,2","objects":1433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12205","reported_seq":12677,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.584964+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:45.584964+0000","last_peered":"2024-04-22T08:32:45.584964+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:45.584964+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2200","ondisk_log_start":"28'2200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30904320,"num_objects":7545,"num_object_clones":0,"num_object_copies":15090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3855},{"shards":"0,2","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12318","reported_seq":12717,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.543543+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:45.543543+0000","last_peered":"2024-04-22T08:32:45.543543+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:45.543543+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31622354,"num_objects":7722,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4177},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12410","reported_seq":12977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:45.513157+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:45.513157+0000","last_peered":"2024-04-22T08:32:45.513157+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:45.513157+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31099300,"num_objects":7596,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3925},{"shards":"0,2","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":501444461,"num_objects":122435,"num_object_clones":0,"num_object_copies":244870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34268,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":122435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171269,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9874,"num_bytes_recovered":33309289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1129746432,"data_stored":1129672187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117065,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668736,"kb_used_data":366392,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93703104,"statfs":{"total":96636764160,"available":95951978496,"internally_reserved":0,"allocated":375185408,"data_stored":374959481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.253}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.23799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920828,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633592,"kb_used_data":309104,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93738248,"statfs":{"total":96636764160,"available":95987965952,"internally_reserved":0,"allocated":316522496,"data_stored":316240905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.2229999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.2619999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":4.2830000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313293,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790380,"kb_used_data":429860,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93581460,"statfs":{"total":96636764160,"available":95827415040,"internally_reserved":0,"allocated":440176640,"data_stored":439961247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.16},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.3360000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":5.3760000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":438812672,"data_stored":438809193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316121088,"data_stored":316057873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374812672,"data_stored":374805121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:52.151 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12420", 'reported_seq': 12906, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.519650+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:45.519650+0000', 'last_peered': '2024-04-22T08:32:45.519650+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:45.519650+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31735808, 'num_objects': 7748, 'num_object_clones': 0, 'num_object_copies': 15496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3584, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12420, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 210, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4164}, {'shards': '0,2', 'objects': 1458}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12237", 'reported_seq': 12586, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.639658+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:45.639658+0000', 'last_peered': '2024-04-22T08:32:45.639658+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:45.639658+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31236120, 'num_objects': 7627, 'num_object_clones': 0, 'num_object_copies': 15254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3645, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7627, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9064, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1452}, {'shards': '2', 'objects': 3983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12439", 'reported_seq': 14093, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.598436+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:45.598436+0000', 'last_peered': '2024-04-22T08:32:45.598436+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:45.598436+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31162368, 'num_objects': 7608, 'num_object_clones': 0, 'num_object_copies': 15216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1672, 'num_objects_misplaced': 1465, 'num_objects_unfound': 0, 'num_objects_dirty': 7608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9308, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 745, 'num_bytes_recovered': 2232320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4471}, {'shards': '0,2', 'objects': 1465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12559", 'reported_seq': 12871, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.660296+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:45.660296+0000', 'last_peered': '2024-04-22T08:32:45.660296+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:45.660296+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2500", 'ondisk_log_start': "28'2500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31384786, 'num_objects': 7664, 'num_object_clones': 0, 'num_object_copies': 15328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12499, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1472}, {'shards': '2', 'objects': 3869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:52.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12417", 'reported_seq': 13011, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.562815+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:45.562815+0000', 'last_peered': '2024-04-22T08:32:45.562815+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:45.562815+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31823058, 'num_objects': 7771, 'num_object_clones': 0, 'num_object_copies': 15542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3516, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12415, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 260, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4257}, {'shards': '0,2', 'objects': 1451}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12254", 'reported_seq': 12645, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.608922+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:45.608922+0000', 'last_peered': '2024-04-22T08:32:45.608922+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:45.608922+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31436800, 'num_objects': 7675, 'num_object_clones': 0, 'num_object_copies': 15350, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3631, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8708, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 121, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1463}, {'shards': '2', 'objects': 4046}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12360", 'reported_seq': 12868, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.617493+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:45.617493+0000', 'last_peered': '2024-04-22T08:32:45.617493+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:45.617493+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31875689, 'num_objects': 7783, 'num_object_clones': 0, 'num_object_copies': 15566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3509, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12359, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 215, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4275}, {'shards': '0,2', 'objects': 1433}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:52.153 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12205", 'reported_seq': 12677, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.584964+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:45.584964+0000', 'last_peered': '2024-04-22T08:32:45.584964+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:45.584964+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30904320, 'num_objects': 7545, 'num_object_clones': 0, 'num_object_copies': 15090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3693, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8436, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 147, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3855}, {'shards': '0,2', 'objects': 1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:52.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12318", 'reported_seq': 12717, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.543543+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:45.543543+0000', 'last_peered': '2024-04-22T08:32:45.543543+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:45.543543+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31622354, 'num_objects': 7722, 'num_object_clones': 0, 'num_object_copies': 15444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3547, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12316, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 162, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4177}, {'shards': '0,2', 'objects': 1408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:52.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12410", 'reported_seq': 12977, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:45.513157+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:45.513157+0000', 'last_peered': '2024-04-22T08:32:45.513157+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:45.513157+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31099300, 'num_objects': 7596, 'num_object_clones': 0, 'num_object_copies': 15192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3674, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12407, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 215, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3925}, {'shards': '0,2', 'objects': 1478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:52.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:53.137 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34197/240930 objects degraded (14.194%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":408,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":120465,"data_bytes":493375341,"bytes_used":2145808384,"bytes_avail":287764484096,"bytes_total":289910292480,"degraded_objects":34197,"degraded_total":240930,"degraded_ratio":0.14193749221765659,"misplaced_objects":1333,"misplaced_total":240930,"misplaced_ratio":0.0055327273481924214,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":104024,"recovering_keys_per_sec":0,"num_objects_recovered":574,"num_bytes_recovered":1249280,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":512},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:53.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:53.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:53.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:53.799 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:53.860 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-22T08:32:52.287480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493375341,"num_objects":120465,"num_object_clones":0,"num_object_copies":240930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34197,"num_objects_misplaced":1333,"num_objects_unfound":0,"num_objects_dirty":120465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173239,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10077,"num_bytes_recovered":33763945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161064,"ondisk_log_size":161064,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2095516,"kb_used_data":1095864,"kb_used_omap":12,"kb_used_meta":999603,"kb_avail":281020004,"statfs":{"total":289910292480,"available":287764484096,"internally_reserved":0,"allocated":1122164736,"data_stored":1121441825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1023593582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,4,5,1,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25214976,"num_objects":-6156,"num_object_clones":0,"num_object_copies":-12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":-408,"num_objects_unfound":0,"num_objects_dirty":-6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":1249280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009426"},"pg_stats":[{"pgid":"2.f","version":"39'12558","reported_seq":13072,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.674399+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:50.674399+0000","last_peered":"2024-04-22T08:32:50.674399+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:50.674399+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31170560,"num_objects":7610,"num_object_clones":0,"num_object_copies":15220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4040},{"shards":"0,2","objects":1320}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12600","reported_seq":12718,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:50.611225+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:50.611225+0000","last_peered":"2024-04-22T08:32:50.611225+0000","last_clean":"2024-04-22T08:32:50.611225+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:50.611225+0000","last_undegraded":"2024-04-22T08:32:50.611225+0000","last_fullsized":"2024-04-22T08:32:50.611225+0000","mapping_epoch":27,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":7632,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12348","reported_seq":12705,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.536454+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:50.536454+0000","last_peered":"2024-04-22T08:32:50.536454+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:50.536454+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30781464,"num_objects":7516,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9175,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1341},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12599","reported_seq":12632,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:50.566977+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:50.566977+0000","last_peered":"2024-04-22T08:32:50.566977+0000","last_clean":"2024-04-22T08:32:50.566977+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:50.566977+0000","last_undegraded":"2024-04-22T08:32:50.566977+0000","last_fullsized":"2024-04-22T08:32:50.566977+0000","mapping_epoch":27,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30781440,"num_objects":7515,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12444","reported_seq":12585,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:50.658212+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:50.658212+0000","last_peered":"2024-04-22T08:32:50.658212+0000","last_clean":"2024-04-22T08:32:50.658212+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:50.658212+0000","last_undegraded":"2024-04-22T08:32:50.658212+0000","last_fullsized":"2024-04-22T08:32:50.658212+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30617600,"num_objects":7475,"num_object_clones":0,"num_object_copies":14950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12562","reported_seq":14438,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.655685+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:50.655685+0000","last_peered":"2024-04-22T08:32:50.655685+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:50.655685+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30658560,"num_objects":7485,"num_object_clones":0,"num_object_copies":14970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1693,"num_objects_misplaced":1333,"num_objects_unfound":0,"num_objects_dirty":7485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":2686976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4459},{"shards":"0,2","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12349","reported_seq":20088,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:49.333154+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:49.333154+0000","last_peered":"2024-04-22T08:32:49.333154+0000","last_clean":"2024-04-22T08:32:49.333154+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:49.333154+0000","last_undegraded":"2024-04-22T08:32:49.333154+0000","last_fullsized":"2024-04-22T08:32:49.333154+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30781440,"num_objects":7515,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8717,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12372","reported_seq":12519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:50.486271+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:50.486271+0000","last_peered":"2024-04-22T08:32:50.486271+0000","last_clean":"2024-04-22T08:32:50.486271+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:50.486271+0000","last_undegraded":"2024-04-22T08:32:50.486271+0000","last_fullsized":"2024-04-22T08:32:50.486271+0000","mapping_epoch":27,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":7434,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12682","reported_seq":13020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.524377+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:50.524377+0000","last_peered":"2024-04-22T08:32:50.524377+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:50.524377+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30880978,"num_objects":7541,"num_object_clones":0,"num_object_copies":15082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12622,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1349},{"shards":"2","objects":3759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12563","reported_seq":13175,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.631489+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:50.631489+0000","last_peered":"2024-04-22T08:32:50.631489+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:50.631489+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31225042,"num_objects":7625,"num_object_clones":0,"num_object_copies":15250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12561,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4120},{"shards":"0,2","objects":1305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12387","reported_seq":12814,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.594976+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:50.594976+0000","last_peered":"2024-04-22T08:32:50.594976+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:50.594976+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30892032,"num_objects":7542,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8841,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1330},{"shards":"2","objects":3931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12464","reported_seq":20141,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:50.583375+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:50.583375+0000","last_peered":"2024-04-22T08:32:50.583375+0000","last_clean":"2024-04-22T08:32:50.583375+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:50.583375+0000","last_undegraded":"2024-04-22T08:32:50.583375+0000","last_fullsized":"2024-04-22T08:32:50.583375+0000","mapping_epoch":38,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30598354,"num_objects":7472,"num_object_clones":0,"num_object_copies":14944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12462,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12472","reported_seq":12997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.529601+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:50.529601+0000","last_peered":"2024-04-22T08:32:50.529601+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:50.529601+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31416937,"num_objects":7671,"num_object_clones":0,"num_object_copies":15342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4172},{"shards":"0,2","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12357","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.677395+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:50.677395+0000","last_peered":"2024-04-22T08:32:50.677395+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:50.677395+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30281728,"num_objects":7393,"num_object_clones":0,"num_object_copies":14786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3716},{"shards":"0,2","objects":1359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12465","reported_seq":12880,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.600849+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:50.600849+0000","last_peered":"2024-04-22T08:32:50.600849+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:50.600849+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31020242,"num_objects":7575,"num_object_clones":0,"num_object_copies":15150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4038},{"shards":"0,2","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12542","reported_seq":13117,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:50.531304+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:50.531304+0000","last_peered":"2024-04-22T08:32:50.531304+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:50.531304+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30558628,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12539,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3797},{"shards":"0,2","objects":1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":493375341,"num_objects":120465,"num_object_clones":0,"num_object_copies":240930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34197,"num_objects_misplaced":1333,"num_objects_unfound":0,"num_objects_dirty":120465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173239,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10077,"num_bytes_recovered":33763945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1119092736,"data_stored":1119018491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161064,"ondisk_log_size":161064,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117066,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669492,"kb_used_data":363040,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93702348,"statfs":{"total":96636764160,"available":95951204352,"internally_reserved":0,"allocated":371752960,"data_stored":371527033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.496},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.255}]}]},{"osd":1,"up_from":33,"seq":141733920828,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633592,"kb_used_data":309104,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93738248,"statfs":{"total":96636764160,"available":95987965952,"internally_reserved":0,"allocated":316522496,"data_stored":316240905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":4.2229999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.2619999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":4.2830000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313294,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792432,"kb_used_data":423720,"kb_used_omap":4,"kb_used_meta":368699,"kb_avail":93579408,"statfs":{"total":96636764160,"available":95825313792,"internally_reserved":0,"allocated":433889280,"data_stored":433673887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":377548499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.798},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":2.835},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":2.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":432316416,"data_stored":432312937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316121088,"data_stored":316057873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370655232,"data_stored":370647681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:56.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:57.441 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:57.473 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-04-22T08:32:56.288450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2067424,"kb_used_data":1084160,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281048096,"statfs":{"total":289910292480,"available":287793250304,"internally_reserved":0,"allocated":1110179840,"data_stored":1109456929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,8,10,6,4,0,1,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26157056,"num_objects":-6386,"num_object_clones":0,"num_object_copies":-12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":-380,"num_objects_unfound":0,"num_objects_dirty":-6386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004834"},"pg_stats":[{"pgid":"2.f","version":"39'12674","reported_seq":13210,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.604518+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:55.604518+0000","last_peered":"2024-04-22T08:32:55.604518+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:55.604518+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30695424,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3935},{"shards":"0,2","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12726","reported_seq":12844,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.624773+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:55.624773+0000","last_peered":"2024-04-22T08:32:55.624773+0000","last_clean":"2024-04-22T08:32:55.624773+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:55.624773+0000","last_undegraded":"2024-04-22T08:32:55.624773+0000","last_fullsized":"2024-04-22T08:32:55.624773+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30744576,"num_objects":7506,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12479","reported_seq":12850,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.692231+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:55.692231+0000","last_peered":"2024-04-22T08:32:55.692231+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:55.692231+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30244888,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9306,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1210},{"shards":"2","objects":3752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12734","reported_seq":12767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.638323+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:55.638323+0000","last_peered":"2024-04-22T08:32:55.638323+0000","last_clean":"2024-04-22T08:32:55.638323+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:55.638323+0000","last_undegraded":"2024-04-22T08:32:55.638323+0000","last_fullsized":"2024-04-22T08:32:55.638323+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30228480,"num_objects":7380,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12578","reported_seq":12719,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.516674+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:55.516674+0000","last_peered":"2024-04-22T08:32:55.516674+0000","last_clean":"2024-04-22T08:32:55.516674+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:55.516674+0000","last_undegraded":"2024-04-22T08:32:55.516674+0000","last_fullsized":"2024-04-22T08:32:55.516674+0000","mapping_epoch":27,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30068736,"num_objects":7341,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9670,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12692","reported_seq":14774,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.666906+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:55.666906+0000","last_peered":"2024-04-22T08:32:55.666906+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:55.666906+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30126080,"num_objects":7355,"num_object_clones":0,"num_object_copies":14710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1710,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":7355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4432},{"shards":"0,2","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12499","reported_seq":20238,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:54.416233+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:54.416233+0000","last_peered":"2024-04-22T08:32:54.416233+0000","last_clean":"2024-04-22T08:32:54.416233+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:54.416233+0000","last_undegraded":"2024-04-22T08:32:54.416233+0000","last_fullsized":"2024-04-22T08:32:54.416233+0000","mapping_epoch":36,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30167040,"num_objects":7365,"num_object_clones":0,"num_object_copies":14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12493","reported_seq":12640,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.655625+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:55.655625+0000","last_peered":"2024-04-22T08:32:55.655625+0000","last_clean":"2024-04-22T08:32:55.655625+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:55.655625+0000","last_undegraded":"2024-04-22T08:32:55.655625+0000","last_fullsized":"2024-04-22T08:32:55.655625+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29954048,"num_objects":7313,"num_object_clones":0,"num_object_copies":14626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12808","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.648641+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:55.648641+0000","last_peered":"2024-04-22T08:32:55.648641+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:55.648641+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30364882,"num_objects":7415,"num_object_clones":0,"num_object_copies":14830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12748,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1223},{"shards":"2","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12695","reported_seq":13339,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.658086+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:55.658086+0000","last_peered":"2024-04-22T08:32:55.658086+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:55.658086+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30684370,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12693,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4004},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12520","reported_seq":12969,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.577148+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:55.577148+0000","last_peered":"2024-04-22T08:32:55.577148+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:55.577148+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347264,"num_objects":7409,"num_object_clones":0,"num_object_copies":14818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1197},{"shards":"2","objects":3809}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12590","reported_seq":20267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.538251+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:55.538251+0000","last_peered":"2024-04-22T08:32:55.538251+0000","last_clean":"2024-04-22T08:32:55.538251+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:55.538251+0000","last_undegraded":"2024-04-22T08:32:55.538251+0000","last_fullsized":"2024-04-22T08:32:55.538251+0000","mapping_epoch":38,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30082258,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12620","reported_seq":13159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.723527+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:55.723527+0000","last_peered":"2024-04-22T08:32:55.723527+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:55.723527+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30810729,"num_objects":7523,"num_object_clones":0,"num_object_copies":15046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4032},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12498","reported_seq":13010,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.659673+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:55.659673+0000","last_peered":"2024-04-22T08:32:55.659673+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:55.659673+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29704192,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12600","reported_seq":13035,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.710377+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:55.710377+0000","last_peered":"2024-04-22T08:32:55.710377+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:55.710377+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30467282,"num_objects":7440,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12598,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3913},{"shards":"0,2","objects":1126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12694","reported_seq":13289,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.700954+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:55.700954+0000","last_peered":"2024-04-22T08:32:55.700954+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:55.700954+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29936036,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12691,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107005440,"data_stored":1106931195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117067,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669460,"kb_used_data":358920,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93702380,"statfs":{"total":96636764160,"available":95951237120,"internally_reserved":0,"allocated":367534080,"data_stored":367308153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.38},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.1059999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":6.1340000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920829,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624316,"kb_used_data":308020,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93747524,"statfs":{"total":96636764160,"available":95997464576,"internally_reserved":0,"allocated":315412480,"data_stored":315130889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":323874508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.53000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313295,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773648,"kb_used_data":417220,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93598192,"statfs":{"total":96636764160,"available":95844548608,"internally_reserved":0,"allocated":427233280,"data_stored":427017887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.5489999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":2.585},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":2.2370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":425619456,"data_stored":425615977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314970112,"data_stored":314906897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366415872,"data_stored":366408321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:57.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:58.173 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:58.254 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-04-22T08:32:56.288450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2067424,"kb_used_data":1084160,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281048096,"statfs":{"total":289910292480,"available":287793250304,"internally_reserved":0,"allocated":1110179840,"data_stored":1109456929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,8,10,6,4,0,1,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26157056,"num_objects":-6386,"num_object_clones":0,"num_object_copies":-12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":-380,"num_objects_unfound":0,"num_objects_dirty":-6386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004834"},"pg_stats":[{"pgid":"2.f","version":"39'12674","reported_seq":13210,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.604518+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:55.604518+0000","last_peered":"2024-04-22T08:32:55.604518+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:55.604518+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30695424,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3935},{"shards":"0,2","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12726","reported_seq":12844,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.624773+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:55.624773+0000","last_peered":"2024-04-22T08:32:55.624773+0000","last_clean":"2024-04-22T08:32:55.624773+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:55.624773+0000","last_undegraded":"2024-04-22T08:32:55.624773+0000","last_fullsized":"2024-04-22T08:32:55.624773+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30744576,"num_objects":7506,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12479","reported_seq":12850,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.692231+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:55.692231+0000","last_peered":"2024-04-22T08:32:55.692231+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:55.692231+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30244888,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9306,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1210},{"shards":"2","objects":3752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12734","reported_seq":12767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.638323+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:55.638323+0000","last_peered":"2024-04-22T08:32:55.638323+0000","last_clean":"2024-04-22T08:32:55.638323+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:55.638323+0000","last_undegraded":"2024-04-22T08:32:55.638323+0000","last_fullsized":"2024-04-22T08:32:55.638323+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30228480,"num_objects":7380,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12578","reported_seq":12719,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.516674+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:55.516674+0000","last_peered":"2024-04-22T08:32:55.516674+0000","last_clean":"2024-04-22T08:32:55.516674+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:55.516674+0000","last_undegraded":"2024-04-22T08:32:55.516674+0000","last_fullsized":"2024-04-22T08:32:55.516674+0000","mapping_epoch":27,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30068736,"num_objects":7341,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9670,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12692","reported_seq":14774,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.666906+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:55.666906+0000","last_peered":"2024-04-22T08:32:55.666906+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:55.666906+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30126080,"num_objects":7355,"num_object_clones":0,"num_object_copies":14710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1710,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":7355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4432},{"shards":"0,2","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12499","reported_seq":20238,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:54.416233+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:54.416233+0000","last_peered":"2024-04-22T08:32:54.416233+0000","last_clean":"2024-04-22T08:32:54.416233+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:54.416233+0000","last_undegraded":"2024-04-22T08:32:54.416233+0000","last_fullsized":"2024-04-22T08:32:54.416233+0000","mapping_epoch":36,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30167040,"num_objects":7365,"num_object_clones":0,"num_object_copies":14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12493","reported_seq":12640,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.655625+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:55.655625+0000","last_peered":"2024-04-22T08:32:55.655625+0000","last_clean":"2024-04-22T08:32:55.655625+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:55.655625+0000","last_undegraded":"2024-04-22T08:32:55.655625+0000","last_fullsized":"2024-04-22T08:32:55.655625+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29954048,"num_objects":7313,"num_object_clones":0,"num_object_copies":14626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12808","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.648641+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:55.648641+0000","last_peered":"2024-04-22T08:32:55.648641+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:55.648641+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30364882,"num_objects":7415,"num_object_clones":0,"num_object_copies":14830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12748,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1223},{"shards":"2","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12695","reported_seq":13339,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.658086+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:55.658086+0000","last_peered":"2024-04-22T08:32:55.658086+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:55.658086+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30684370,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12693,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4004},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12520","reported_seq":12969,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.577148+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:55.577148+0000","last_peered":"2024-04-22T08:32:55.577148+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:55.577148+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347264,"num_objects":7409,"num_object_clones":0,"num_object_copies":14818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1197},{"shards":"2","objects":3809}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12590","reported_seq":20267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.538251+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:55.538251+0000","last_peered":"2024-04-22T08:32:55.538251+0000","last_clean":"2024-04-22T08:32:55.538251+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:55.538251+0000","last_undegraded":"2024-04-22T08:32:55.538251+0000","last_fullsized":"2024-04-22T08:32:55.538251+0000","mapping_epoch":38,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30082258,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12620","reported_seq":13159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.723527+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:55.723527+0000","last_peered":"2024-04-22T08:32:55.723527+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:55.723527+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30810729,"num_objects":7523,"num_object_clones":0,"num_object_copies":15046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4032},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12498","reported_seq":13010,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.659673+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:55.659673+0000","last_peered":"2024-04-22T08:32:55.659673+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:55.659673+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29704192,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12600","reported_seq":13035,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.710377+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:55.710377+0000","last_peered":"2024-04-22T08:32:55.710377+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:55.710377+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30467282,"num_objects":7440,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12598,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3913},{"shards":"0,2","objects":1126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12694","reported_seq":13289,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.700954+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:55.700954+0000","last_peered":"2024-04-22T08:32:55.700954+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:55.700954+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29936036,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12691,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107005440,"data_stored":1106931195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117067,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669460,"kb_used_data":358920,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93702380,"statfs":{"total":96636764160,"available":95951237120,"internally_reserved":0,"allocated":367534080,"data_stored":367308153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.38},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.1059999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":6.1340000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920829,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624316,"kb_used_data":308020,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93747524,"statfs":{"total":96636764160,"available":95997464576,"internally_reserved":0,"allocated":315412480,"data_stored":315130889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":323874508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.53000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313295,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773648,"kb_used_data":417220,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93598192,"statfs":{"total":96636764160,"available":95844548608,"internally_reserved":0,"allocated":427233280,"data_stored":427017887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.5489999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":2.585},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":2.2370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":425619456,"data_stored":425615977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314970112,"data_stored":314906897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366415872,"data_stored":366408321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12674", 'reported_seq': 13210, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.604518+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:32:55.604518+0000', 'last_peered': '2024-04-22T08:32:55.604518+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:32:55.604518+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30695424, 'num_objects': 7494, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3559, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12674, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 235, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3935}, {'shards': '0,2', 'objects': 1204}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12479", 'reported_seq': 12850, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.692231+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:32:55.692231+0000', 'last_peered': '2024-04-22T08:32:55.692231+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:32:55.692231+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30244888, 'num_objects': 7385, 'num_object_clones': 0, 'num_object_copies': 14770, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3634, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7385, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9306, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1210}, {'shards': '2', 'objects': 3752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12692", 'reported_seq': 14774, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.666906+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:32:55.666906+0000', 'last_peered': '2024-04-22T08:32:55.666906+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:32:55.666906+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30126080, 'num_objects': 7355, 'num_object_clones': 0, 'num_object_copies': 14710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1710, 'num_objects_misplaced': 1213, 'num_objects_unfound': 0, 'num_objects_dirty': 7355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9561, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 959, 'num_bytes_recovered': 3096576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4432}, {'shards': '0,2', 'objects': 1213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:32:58.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12808", 'reported_seq': 13153, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.648641+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:32:55.648641+0000', 'last_peered': '2024-04-22T08:32:55.648641+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:32:55.648641+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30364882, 'num_objects': 7415, 'num_object_clones': 0, 'num_object_copies': 14830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3780, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12748, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 115, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1223}, {'shards': '2', 'objects': 3637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12695", 'reported_seq': 13339, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.658086+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:32:55.658086+0000', 'last_peered': '2024-04-22T08:32:55.658086+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:32:55.658086+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30684370, 'num_objects': 7493, 'num_object_clones': 0, 'num_object_copies': 14986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3491, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12693, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 285, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4004}, {'shards': '0,2', 'objects': 1173}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12520", 'reported_seq': 12969, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.577148+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:32:55.577148+0000', 'last_peered': '2024-04-22T08:32:55.577148+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:32:55.577148+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2500", 'ondisk_log_start': "28'2500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30347264, 'num_objects': 7409, 'num_object_clones': 0, 'num_object_copies': 14818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3602, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8974, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 150, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1197}, {'shards': '2', 'objects': 3809}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12620", 'reported_seq': 13159, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.723527+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:32:55.723527+0000', 'last_peered': '2024-04-22T08:32:55.723527+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:32:55.723527+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30810729, 'num_objects': 7523, 'num_object_clones': 0, 'num_object_copies': 15046, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3492, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12619, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 232, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4032}, {'shards': '0,2', 'objects': 1173}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12498", 'reported_seq': 13010, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.659673+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:32:55.659673+0000', 'last_peered': '2024-04-22T08:32:55.659673+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:32:55.659673+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29704192, 'num_objects': 7252, 'num_object_clones': 0, 'num_object_copies': 14504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3673, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8729, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 167, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3582}, {'shards': '0,2', 'objects': 1218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:32:58.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12600", 'reported_seq': 13035, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.710377+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:32:55.710377+0000', 'last_peered': '2024-04-22T08:32:55.710377+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:32:55.710377+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2500", 'ondisk_log_start': "28'2500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30467282, 'num_objects': 7440, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12598, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 180, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3913}, {'shards': '0,2', 'objects': 1126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.258 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:32:58.258 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12694", 'reported_seq': 13289, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:32:55.700954+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:32:55.700954+0000', 'last_peered': '2024-04-22T08:32:55.700954+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:32:55.700954+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29936036, 'num_objects': 7312, 'num_object_clones': 0, 'num_object_copies': 14624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3660, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12691, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 229, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3655}, {'shards': '0,2', 'objects': 1194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:32:58.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:32:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:59.060 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34130/236658 objects degraded (14.422%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":413,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":118329,"data_bytes":484626285,"bytes_used":2117042176,"bytes_avail":287793250304,"bytes_total":289910292480,"degraded_objects":34130,"degraded_total":236658,"degraded_ratio":0.14421654877502557,"misplaced_objects":1213,"misplaced_total":236658,"misplaced_ratio":0.0051255398085000292,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":71991,"recovering_keys_per_sec":0,"num_objects_recovered":390,"num_bytes_recovered":864256,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":352},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:32:59.131 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:32:59.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:32:59.890 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:32:59.892 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:32:59.922 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-04-22T08:32:58.288999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2067424,"kb_used_data":1084160,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281048096,"statfs":{"total":289910292480,"available":287793250304,"internally_reserved":0,"allocated":1110179840,"data_stored":1109456929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,8,10,6,4,0,1,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17309696,"num_objects":-4226,"num_object_clones":0,"num_object_copies":-8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":-252,"num_objects_unfound":0,"num_objects_dirty":-4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":864256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004927"},"pg_stats":[{"pgid":"2.f","version":"39'12674","reported_seq":13210,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.604518+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:32:55.604518+0000","last_peered":"2024-04-22T08:32:55.604518+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:32:55.604518+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30695424,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3935},{"shards":"0,2","objects":1204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12726","reported_seq":12844,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.624773+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:32:55.624773+0000","last_peered":"2024-04-22T08:32:55.624773+0000","last_clean":"2024-04-22T08:32:55.624773+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:32:55.624773+0000","last_undegraded":"2024-04-22T08:32:55.624773+0000","last_fullsized":"2024-04-22T08:32:55.624773+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30744576,"num_objects":7506,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12479","reported_seq":12850,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.692231+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:32:55.692231+0000","last_peered":"2024-04-22T08:32:55.692231+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:32:55.692231+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30244888,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9306,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1210},{"shards":"2","objects":3752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12734","reported_seq":12767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.638323+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:32:55.638323+0000","last_peered":"2024-04-22T08:32:55.638323+0000","last_clean":"2024-04-22T08:32:55.638323+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:32:55.638323+0000","last_undegraded":"2024-04-22T08:32:55.638323+0000","last_fullsized":"2024-04-22T08:32:55.638323+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30228480,"num_objects":7380,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12578","reported_seq":12719,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.516674+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:32:55.516674+0000","last_peered":"2024-04-22T08:32:55.516674+0000","last_clean":"2024-04-22T08:32:55.516674+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:32:55.516674+0000","last_undegraded":"2024-04-22T08:32:55.516674+0000","last_fullsized":"2024-04-22T08:32:55.516674+0000","mapping_epoch":27,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30068736,"num_objects":7341,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9670,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12692","reported_seq":14774,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.666906+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:32:55.666906+0000","last_peered":"2024-04-22T08:32:55.666906+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:32:55.666906+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30126080,"num_objects":7355,"num_object_clones":0,"num_object_copies":14710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1710,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":7355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4432},{"shards":"0,2","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12499","reported_seq":20238,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:54.416233+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:54.416233+0000","last_peered":"2024-04-22T08:32:54.416233+0000","last_clean":"2024-04-22T08:32:54.416233+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:54.416233+0000","last_undegraded":"2024-04-22T08:32:54.416233+0000","last_fullsized":"2024-04-22T08:32:54.416233+0000","mapping_epoch":36,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30167040,"num_objects":7365,"num_object_clones":0,"num_object_copies":14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12493","reported_seq":12640,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.655625+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:32:55.655625+0000","last_peered":"2024-04-22T08:32:55.655625+0000","last_clean":"2024-04-22T08:32:55.655625+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:32:55.655625+0000","last_undegraded":"2024-04-22T08:32:55.655625+0000","last_fullsized":"2024-04-22T08:32:55.655625+0000","mapping_epoch":27,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29954048,"num_objects":7313,"num_object_clones":0,"num_object_copies":14626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12808","reported_seq":13153,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.648641+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:32:55.648641+0000","last_peered":"2024-04-22T08:32:55.648641+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:32:55.648641+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30364882,"num_objects":7415,"num_object_clones":0,"num_object_copies":14830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12748,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1223},{"shards":"2","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12695","reported_seq":13339,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.658086+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:32:55.658086+0000","last_peered":"2024-04-22T08:32:55.658086+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:32:55.658086+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30684370,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12693,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4004},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12520","reported_seq":12969,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.577148+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:32:55.577148+0000","last_peered":"2024-04-22T08:32:55.577148+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:32:55.577148+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30347264,"num_objects":7409,"num_object_clones":0,"num_object_copies":14818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1197},{"shards":"2","objects":3809}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12590","reported_seq":20267,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:55.538251+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:32:55.538251+0000","last_peered":"2024-04-22T08:32:55.538251+0000","last_clean":"2024-04-22T08:32:55.538251+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:32:55.538251+0000","last_undegraded":"2024-04-22T08:32:55.538251+0000","last_fullsized":"2024-04-22T08:32:55.538251+0000","mapping_epoch":38,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30082258,"num_objects":7346,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12620","reported_seq":13159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.723527+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:32:55.723527+0000","last_peered":"2024-04-22T08:32:55.723527+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:32:55.723527+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30810729,"num_objects":7523,"num_object_clones":0,"num_object_copies":15046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4032},{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12498","reported_seq":13010,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.659673+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:32:55.659673+0000","last_peered":"2024-04-22T08:32:55.659673+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:32:55.659673+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2400","ondisk_log_start":"28'2400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29704192,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3582},{"shards":"0,2","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12600","reported_seq":13035,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.710377+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:32:55.710377+0000","last_peered":"2024-04-22T08:32:55.710377+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:32:55.710377+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2500","ondisk_log_start":"28'2500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30467282,"num_objects":7440,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12598,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3913},{"shards":"0,2","objects":1126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12694","reported_seq":13289,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:32:55.700954+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:32:55.700954+0000","last_peered":"2024-04-22T08:32:55.700954+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:32:55.700954+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29936036,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12691,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484626285,"num_objects":118329,"num_object_clones":0,"num_object_copies":236658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34130,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":118329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175375,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10264,"num_bytes_recovered":34173545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107005440,"data_stored":1106931195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161100,"ondisk_log_size":161100,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117067,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669460,"kb_used_data":358920,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93702380,"statfs":{"total":96636764160,"available":95951237120,"internally_reserved":0,"allocated":367534080,"data_stored":367308153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.38},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.1059999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":6.1340000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920829,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624316,"kb_used_data":308020,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93747524,"statfs":{"total":96636764160,"available":95997464576,"internally_reserved":0,"allocated":315412480,"data_stored":315130889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":323874508},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.53000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313295,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773648,"kb_used_data":417220,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93598192,"statfs":{"total":96636764160,"available":95844548608,"internally_reserved":0,"allocated":427233280,"data_stored":427017887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":364965587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.018,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.5819999999999999,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.5489999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":1.1299999999999999,"15min":0.96199999999999997},"min":{"1min":0.20599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.8169999999999999,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":2.585},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":2.2370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":425619456,"data_stored":425615977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314970112,"data_stored":314906897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366415872,"data_stored":366408321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:02.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:03.671 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:03.673 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:03.734 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-04-22T08:33:02.291478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2055816,"kb_used_data":1072556,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281059704,"statfs":{"total":289910292480,"available":287805136896,"internally_reserved":0,"allocated":1098297344,"data_stored":1097574433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,10,4,7,0,0,4,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25382912,"num_objects":-6197,"num_object_clones":0,"num_object_copies":-12394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":-394,"num_objects_unfound":0,"num_objects_dirty":-6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":1323008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006447"},"pg_stats":[{"pgid":"2.f","version":"39'12808","reported_seq":13358,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.660457+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:00.660457+0000","last_peered":"2024-04-22T08:33:00.660457+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:00.660457+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3808},{"shards":"0,2","objects":1070}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12856","reported_seq":12974,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.610912+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:00.610912+0000","last_peered":"2024-04-22T08:33:00.610912+0000","last_clean":"2024-04-22T08:33:00.610912+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:00.610912+0000","last_undegraded":"2024-04-22T08:33:00.610912+0000","last_fullsized":"2024-04-22T08:33:00.610912+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":7376,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12625","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.711625+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:00.711625+0000","last_peered":"2024-04-22T08:33:00.711625+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:00.711625+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29646872,"num_objects":7239,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9452,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":3617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12863","reported_seq":12896,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.501146+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:00.501146+0000","last_peered":"2024-04-22T08:33:00.501146+0000","last_clean":"2024-04-22T08:33:00.501146+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:00.501146+0000","last_undegraded":"2024-04-22T08:33:00.501146+0000","last_fullsized":"2024-04-22T08:33:00.501146+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700096,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12698","reported_seq":12839,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.781742+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:00.781742+0000","last_peered":"2024-04-22T08:33:00.781742+0000","last_clean":"2024-04-22T08:33:00.781742+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:00.781742+0000","last_undegraded":"2024-04-22T08:33:00.781742+0000","last_fullsized":"2024-04-22T08:33:00.781742+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29577216,"num_objects":7221,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12830","reported_seq":15140,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.772423+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:00.772423+0000","last_peered":"2024-04-22T08:33:00.772423+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:00.772423+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29560832,"num_objects":7217,"num_object_clones":0,"num_object_copies":14434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1738,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":7217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1073,"num_bytes_recovered":3555328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4408},{"shards":"0,2","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12631","reported_seq":20370,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:59.540376+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:59.540376+0000","last_peered":"2024-04-22T08:32:59.540376+0000","last_clean":"2024-04-22T08:32:59.540376+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:59.540376+0000","last_undegraded":"2024-04-22T08:32:59.540376+0000","last_fullsized":"2024-04-22T08:32:59.540376+0000","mapping_epoch":36,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29626368,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12613","reported_seq":12760,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.653739+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:00.653739+0000","last_peered":"2024-04-22T08:33:00.653739+0000","last_clean":"2024-04-22T08:33:00.653739+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:00.653739+0000","last_undegraded":"2024-04-22T08:33:00.653739+0000","last_fullsized":"2024-04-22T08:33:00.653739+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29462528,"num_objects":7193,"num_object_clones":0,"num_object_copies":14386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12955","reported_seq":13321,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.698199+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:00.698199+0000","last_peered":"2024-04-22T08:33:00.698199+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:00.698199+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29762770,"num_objects":7268,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12814","reported_seq":13477,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.771148+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:00.771148+0000","last_peered":"2024-04-22T08:33:00.771148+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:00.771148+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30196946,"num_objects":7374,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12812,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895},{"shards":"0,2","objects":1054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12641","reported_seq":13096,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.693895+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:00.693895+0000","last_peered":"2024-04-22T08:33:00.693895+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:00.693895+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29851648,"num_objects":7288,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3691}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12723","reported_seq":20400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.637753+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:00.637753+0000","last_peered":"2024-04-22T08:33:00.637753+0000","last_clean":"2024-04-22T08:33:00.637753+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:00.637753+0000","last_undegraded":"2024-04-22T08:33:00.637753+0000","last_fullsized":"2024-04-22T08:33:00.637753+0000","mapping_epoch":38,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29537490,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12721,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12751","reported_seq":13310,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.807598+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:00.807598+0000","last_peered":"2024-04-22T08:33:00.807598+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:00.807598+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274153,"num_objects":7392,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3910},{"shards":"0,2","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12631","reported_seq":13157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.719555+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:00.719555+0000","last_peered":"2024-04-22T08:33:00.719555+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:00.719555+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29159424,"num_objects":7119,"num_object_clones":0,"num_object_copies":14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12719","reported_seq":13162,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.734127+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:00.734127+0000","last_peered":"2024-04-22T08:33:00.734127+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:00.734127+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979858,"num_objects":7321,"num_object_clones":0,"num_object_copies":14642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12717,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3798},{"shards":"0,2","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12833","reported_seq":13442,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.695413+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:00.695413+0000","last_peered":"2024-04-22T08:33:00.695413+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:00.695413+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29366692,"num_objects":7173,"num_object_clones":0,"num_object_copies":14346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095290880,"data_stored":1095216635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117068,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653024,"kb_used_data":354776,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93718816,"statfs":{"total":96636764160,"available":95968067584,"internally_reserved":0,"allocated":363290624,"data_stored":363064697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.514},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.28999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920830,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631396,"kb_used_data":306908,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93740444,"statfs":{"total":96636764160,"available":95990214656,"internally_reserved":0,"allocated":314273792,"data_stored":313992201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313296,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771396,"kb_used_data":410872,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93600444,"statfs":{"total":96636764160,"available":95846854656,"internally_reserved":0,"allocated":420732928,"data_stored":420517535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,7,2,2,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.316},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":12.131}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419270656,"data_stored":419267177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313868288,"data_stored":313805073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362151936,"data_stored":362144385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:03.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:04.339 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:04.340 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:04.392 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-04-22T08:33:04.292021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2055816,"kb_used_data":1072556,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281059704,"statfs":{"total":289910292480,"available":287805136896,"internally_reserved":0,"allocated":1098297344,"data_stored":1097574433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,10,4,7,0,0,4,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17313792,"num_objects":-4227,"num_object_clones":0,"num_object_copies":-8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":-262,"num_objects_unfound":0,"num_objects_dirty":-4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004540"},"pg_stats":[{"pgid":"2.f","version":"39'12808","reported_seq":13358,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.660457+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:00.660457+0000","last_peered":"2024-04-22T08:33:00.660457+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:00.660457+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3808},{"shards":"0,2","objects":1070}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12856","reported_seq":12974,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.610912+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:00.610912+0000","last_peered":"2024-04-22T08:33:00.610912+0000","last_clean":"2024-04-22T08:33:00.610912+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:00.610912+0000","last_undegraded":"2024-04-22T08:33:00.610912+0000","last_fullsized":"2024-04-22T08:33:00.610912+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":7376,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12625","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.711625+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:00.711625+0000","last_peered":"2024-04-22T08:33:00.711625+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:00.711625+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29646872,"num_objects":7239,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9452,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":3617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12863","reported_seq":12896,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.501146+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:00.501146+0000","last_peered":"2024-04-22T08:33:00.501146+0000","last_clean":"2024-04-22T08:33:00.501146+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:00.501146+0000","last_undegraded":"2024-04-22T08:33:00.501146+0000","last_fullsized":"2024-04-22T08:33:00.501146+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700096,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12698","reported_seq":12839,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.781742+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:00.781742+0000","last_peered":"2024-04-22T08:33:00.781742+0000","last_clean":"2024-04-22T08:33:00.781742+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:00.781742+0000","last_undegraded":"2024-04-22T08:33:00.781742+0000","last_fullsized":"2024-04-22T08:33:00.781742+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29577216,"num_objects":7221,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12830","reported_seq":15140,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.772423+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:00.772423+0000","last_peered":"2024-04-22T08:33:00.772423+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:00.772423+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29560832,"num_objects":7217,"num_object_clones":0,"num_object_copies":14434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1738,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":7217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1073,"num_bytes_recovered":3555328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4408},{"shards":"0,2","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12631","reported_seq":20370,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:59.540376+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:59.540376+0000","last_peered":"2024-04-22T08:32:59.540376+0000","last_clean":"2024-04-22T08:32:59.540376+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:59.540376+0000","last_undegraded":"2024-04-22T08:32:59.540376+0000","last_fullsized":"2024-04-22T08:32:59.540376+0000","mapping_epoch":36,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29626368,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12613","reported_seq":12760,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.653739+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:00.653739+0000","last_peered":"2024-04-22T08:33:00.653739+0000","last_clean":"2024-04-22T08:33:00.653739+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:00.653739+0000","last_undegraded":"2024-04-22T08:33:00.653739+0000","last_fullsized":"2024-04-22T08:33:00.653739+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29462528,"num_objects":7193,"num_object_clones":0,"num_object_copies":14386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12955","reported_seq":13321,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.698199+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:00.698199+0000","last_peered":"2024-04-22T08:33:00.698199+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:00.698199+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29762770,"num_objects":7268,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12814","reported_seq":13477,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.771148+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:00.771148+0000","last_peered":"2024-04-22T08:33:00.771148+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:00.771148+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30196946,"num_objects":7374,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12812,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895},{"shards":"0,2","objects":1054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12641","reported_seq":13096,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.693895+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:00.693895+0000","last_peered":"2024-04-22T08:33:00.693895+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:00.693895+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29851648,"num_objects":7288,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3691}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12723","reported_seq":20400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.637753+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:00.637753+0000","last_peered":"2024-04-22T08:33:00.637753+0000","last_clean":"2024-04-22T08:33:00.637753+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:00.637753+0000","last_undegraded":"2024-04-22T08:33:00.637753+0000","last_fullsized":"2024-04-22T08:33:00.637753+0000","mapping_epoch":38,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29537490,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12721,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12751","reported_seq":13310,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.807598+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:00.807598+0000","last_peered":"2024-04-22T08:33:00.807598+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:00.807598+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274153,"num_objects":7392,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3910},{"shards":"0,2","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12631","reported_seq":13157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.719555+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:00.719555+0000","last_peered":"2024-04-22T08:33:00.719555+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:00.719555+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29159424,"num_objects":7119,"num_object_clones":0,"num_object_copies":14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12719","reported_seq":13162,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.734127+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:00.734127+0000","last_peered":"2024-04-22T08:33:00.734127+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:00.734127+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979858,"num_objects":7321,"num_object_clones":0,"num_object_copies":14642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12717,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3798},{"shards":"0,2","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12833","reported_seq":13442,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.695413+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:00.695413+0000","last_peered":"2024-04-22T08:33:00.695413+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:00.695413+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29366692,"num_objects":7173,"num_object_clones":0,"num_object_copies":14346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095290880,"data_stored":1095216635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117068,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653024,"kb_used_data":354776,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93718816,"statfs":{"total":96636764160,"available":95968067584,"internally_reserved":0,"allocated":363290624,"data_stored":363064697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.514},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.28999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920830,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631396,"kb_used_data":306908,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93740444,"statfs":{"total":96636764160,"available":95990214656,"internally_reserved":0,"allocated":314273792,"data_stored":313992201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313296,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771396,"kb_used_data":410872,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93600444,"statfs":{"total":96636764160,"available":95846854656,"internally_reserved":0,"allocated":420732928,"data_stored":420517535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,7,2,2,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.316},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":12.131}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419270656,"data_stored":419267177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313868288,"data_stored":313805073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362151936,"data_stored":362144385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:04.393 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:04.393 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12808", 'reported_seq': 13358, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.660457+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:00.660457+0000', 'last_peered': '2024-04-22T08:33:00.660457+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:00.660457+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30146560, 'num_objects': 7360, 'num_object_clones': 0, 'num_object_copies': 14720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12808, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 242, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3808}, {'shards': '0,2', 'objects': 1070}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12625", 'reported_seq': 13016, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.711625+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:00.711625+0000', 'last_peered': '2024-04-22T08:33:00.711625+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:00.711625+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29646872, 'num_objects': 7239, 'num_object_clones': 0, 'num_object_copies': 14478, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3623, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7239, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9452, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 133, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1064}, {'shards': '2', 'objects': 3617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12830", 'reported_seq': 15140, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.772423+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:00.772423+0000', 'last_peered': '2024-04-22T08:33:00.772423+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:00.772423+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29560832, 'num_objects': 7217, 'num_object_clones': 0, 'num_object_copies': 14434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1738, 'num_objects_misplaced': 1071, 'num_objects_unfound': 0, 'num_objects_dirty': 7217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9699, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1073, 'num_bytes_recovered': 3555328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4408}, {'shards': '0,2', 'objects': 1071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'12955", 'reported_seq': 13321, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.698199+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:00.698199+0000', 'last_peered': '2024-04-22T08:33:00.698199+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:00.698199+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29762770, 'num_objects': 7268, 'num_object_clones': 0, 'num_object_copies': 14536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3770, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12895, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 125, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1076}, {'shards': '2', 'objects': 3500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:04.394 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12814", 'reported_seq': 13477, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.771148+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:00.771148+0000', 'last_peered': '2024-04-22T08:33:00.771148+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:00.771148+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30196946, 'num_objects': 7374, 'num_object_clones': 0, 'num_object_copies': 14748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3481, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12812, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 295, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3895}, {'shards': '0,2', 'objects': 1054}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12641", 'reported_seq': 13096, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.693895+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:00.693895+0000', 'last_peered': '2024-04-22T08:33:00.693895+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:00.693895+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29851648, 'num_objects': 7288, 'num_object_clones': 0, 'num_object_copies': 14576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9095, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 153, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1076}, {'shards': '2', 'objects': 3691}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12751", 'reported_seq': 13310, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.807598+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:00.807598+0000', 'last_peered': '2024-04-22T08:33:00.807598+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:00.807598+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2700", 'ondisk_log_start': "28'2700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30274153, 'num_objects': 7392, 'num_object_clones': 0, 'num_object_copies': 14784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3483, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12750, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 241, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3910}, {'shards': '0,2', 'objects': 1042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12631", 'reported_seq': 13157, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.719555+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:00.719555+0000', 'last_peered': '2024-04-22T08:33:00.719555+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:00.719555+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2600", 'ondisk_log_start': "28'2600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29159424, 'num_objects': 7119, 'num_object_clones': 0, 'num_object_copies': 14238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3666, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8862, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 174, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3456}, {'shards': '0,2', 'objects': 1085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:04.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12719", 'reported_seq': 13162, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.734127+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:00.734127+0000', 'last_peered': '2024-04-22T08:33:00.734127+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:00.734127+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2700", 'ondisk_log_start': "28'2700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29979858, 'num_objects': 7321, 'num_object_clones': 0, 'num_object_copies': 14642, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12717, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 184, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3798}, {'shards': '0,2', 'objects': 1007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:04.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12833", 'reported_seq': 13442, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:00.695413+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:00.695413+0000', 'last_peered': '2024-04-22T08:33:00.695413+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:00.695413+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29366692, 'num_objects': 7173, 'num_object_clones': 0, 'num_object_copies': 14346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3653, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12830, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 236, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3523}, {'shards': '0,2', 'objects': 1055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:04.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:05.165 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:05.165 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34090/232476 objects degraded (14.664%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":420,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":116238,"data_bytes":476061549,"bytes_used":2105155584,"bytes_avail":287805136896,"bytes_total":289910292480,"degraded_objects":34090,"degraded_total":232476,"degraded_ratio":0.14663879282162459,"misplaced_objects":1071,"misplaced_total":232476,"misplaced_ratio":0.0046069271666752693,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":72335,"recovering_keys_per_sec":0,"num_objects_recovered":369,"num_bytes_recovered":868352,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":352},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:05.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:05.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:05.974 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:05.975 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:06.025 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-04-22T08:33:04.292021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2055816,"kb_used_data":1072556,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281059704,"statfs":{"total":289910292480,"available":287805136896,"internally_reserved":0,"allocated":1098297344,"data_stored":1097574433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,10,4,7,0,0,4,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17313792,"num_objects":-4227,"num_object_clones":0,"num_object_copies":-8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":-262,"num_objects_unfound":0,"num_objects_dirty":-4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004540"},"pg_stats":[{"pgid":"2.f","version":"39'12808","reported_seq":13358,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.660457+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:00.660457+0000","last_peered":"2024-04-22T08:33:00.660457+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:00.660457+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12808,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3808},{"shards":"0,2","objects":1070}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12856","reported_seq":12974,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.610912+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:00.610912+0000","last_peered":"2024-04-22T08:33:00.610912+0000","last_clean":"2024-04-22T08:33:00.610912+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:00.610912+0000","last_undegraded":"2024-04-22T08:33:00.610912+0000","last_fullsized":"2024-04-22T08:33:00.610912+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":7376,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12625","reported_seq":13016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.711625+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:00.711625+0000","last_peered":"2024-04-22T08:33:00.711625+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:00.711625+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29646872,"num_objects":7239,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9452,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":3617}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'12863","reported_seq":12896,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.501146+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:00.501146+0000","last_peered":"2024-04-22T08:33:00.501146+0000","last_clean":"2024-04-22T08:33:00.501146+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:00.501146+0000","last_undegraded":"2024-04-22T08:33:00.501146+0000","last_fullsized":"2024-04-22T08:33:00.501146+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700096,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12863,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12698","reported_seq":12839,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.781742+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:00.781742+0000","last_peered":"2024-04-22T08:33:00.781742+0000","last_clean":"2024-04-22T08:33:00.781742+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:00.781742+0000","last_undegraded":"2024-04-22T08:33:00.781742+0000","last_fullsized":"2024-04-22T08:33:00.781742+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29577216,"num_objects":7221,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12830","reported_seq":15140,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.772423+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:00.772423+0000","last_peered":"2024-04-22T08:33:00.772423+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:00.772423+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29560832,"num_objects":7217,"num_object_clones":0,"num_object_copies":14434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1738,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":7217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1073,"num_bytes_recovered":3555328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4408},{"shards":"0,2","objects":1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12631","reported_seq":20370,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:59.540376+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:32:59.540376+0000","last_peered":"2024-04-22T08:32:59.540376+0000","last_clean":"2024-04-22T08:32:59.540376+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:32:59.540376+0000","last_undegraded":"2024-04-22T08:32:59.540376+0000","last_fullsized":"2024-04-22T08:32:59.540376+0000","mapping_epoch":36,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29626368,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12613","reported_seq":12760,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.653739+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:00.653739+0000","last_peered":"2024-04-22T08:33:00.653739+0000","last_clean":"2024-04-22T08:33:00.653739+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:00.653739+0000","last_undegraded":"2024-04-22T08:33:00.653739+0000","last_fullsized":"2024-04-22T08:33:00.653739+0000","mapping_epoch":27,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29462528,"num_objects":7193,"num_object_clones":0,"num_object_copies":14386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'12955","reported_seq":13321,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.698199+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:00.698199+0000","last_peered":"2024-04-22T08:33:00.698199+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:00.698199+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29762770,"num_objects":7268,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12814","reported_seq":13477,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.771148+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:00.771148+0000","last_peered":"2024-04-22T08:33:00.771148+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:00.771148+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30196946,"num_objects":7374,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12812,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895},{"shards":"0,2","objects":1054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126880+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:18.126880+0000","last_peered":"2024-04-22T08:32:18.126880+0000","last_clean":"2024-04-22T08:32:18.126880+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:18.126880+0000","last_undegraded":"2024-04-22T08:32:18.126880+0000","last_fullsized":"2024-04-22T08:32:18.126880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12641","reported_seq":13096,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.693895+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:00.693895+0000","last_peered":"2024-04-22T08:33:00.693895+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:00.693895+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29851648,"num_objects":7288,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1076},{"shards":"2","objects":3691}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12723","reported_seq":20400,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:00.637753+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:00.637753+0000","last_peered":"2024-04-22T08:33:00.637753+0000","last_clean":"2024-04-22T08:33:00.637753+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:00.637753+0000","last_undegraded":"2024-04-22T08:33:00.637753+0000","last_fullsized":"2024-04-22T08:33:00.637753+0000","mapping_epoch":38,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29537490,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12721,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12751","reported_seq":13310,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.807598+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:00.807598+0000","last_peered":"2024-04-22T08:33:00.807598+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:00.807598+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274153,"num_objects":7392,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12750,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3910},{"shards":"0,2","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12631","reported_seq":13157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.719555+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:00.719555+0000","last_peered":"2024-04-22T08:33:00.719555+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:00.719555+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2600","ondisk_log_start":"28'2600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29159424,"num_objects":7119,"num_object_clones":0,"num_object_copies":14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3456},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12719","reported_seq":13162,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.734127+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:00.734127+0000","last_peered":"2024-04-22T08:33:00.734127+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:00.734127+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979858,"num_objects":7321,"num_object_clones":0,"num_object_copies":14642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12717,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3798},{"shards":"0,2","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12833","reported_seq":13442,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:00.695413+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:00.695413+0000","last_peered":"2024-04-22T08:33:00.695413+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:00.695413+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29366692,"num_objects":7173,"num_object_clones":0,"num_object_copies":14346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12830,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":476061549,"num_objects":116238,"num_object_clones":0,"num_object_copies":232476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34090,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":116238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177466,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10446,"num_bytes_recovered":34632297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095290880,"data_stored":1095216635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160591,"ondisk_log_size":160591,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117068,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653024,"kb_used_data":354776,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93718816,"statfs":{"total":96636764160,"available":95968067584,"internally_reserved":0,"allocated":363290624,"data_stored":363064697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.514},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.28999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920830,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631396,"kb_used_data":306908,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93740444,"statfs":{"total":96636764160,"available":95990214656,"internally_reserved":0,"allocated":314273792,"data_stored":313992201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":332263116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76700000000000002,"15min":0.58399999999999996},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3410000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.82199999999999995,"15min":0.92200000000000004},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.1929999999999996,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.6399999999999999,"15min":1.462},"min":{"1min":0.24199999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.1340000000000003,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.456,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.1669999999999998,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313296,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771396,"kb_used_data":410872,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93600444,"statfs":{"total":96636764160,"available":95846854656,"internally_reserved":0,"allocated":420732928,"data_stored":420517535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":369159891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,7,2,2,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.316},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":12.131}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419270656,"data_stored":419267177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313868288,"data_stored":313805073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362151936,"data_stored":362144385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:09.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:09.690 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:09.730 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-04-22T08:33:08.295481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":467414893,"num_objects":114127,"num_object_clones":0,"num_object_copies":228254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":114127,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179577,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160802,"ondisk_log_size":160802,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2044204,"kb_used_data":1060944,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281071316,"statfs":{"total":289910292480,"available":287817027584,"internally_reserved":0,"allocated":1086406656,"data_stored":1085683745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,7,3,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17211392,"num_objects":-4202,"num_object_clones":0,"num_object_copies":-8404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":377,"num_bytes_recovered":929792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007030"},"pg_stats":[{"pgid":"2.f","version":"39'12943","reported_seq":13513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.767937+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:05.767937+0000","last_peered":"2024-04-22T08:33:05.767937+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:05.767937+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29593600,"num_objects":7225,"num_object_clones":0,"num_object_copies":14450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12943,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3683},{"shards":"0,2","objects":935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12986","reported_seq":13104,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.603738+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:05.603738+0000","last_peered":"2024-04-22T08:33:05.603738+0000","last_clean":"2024-04-22T08:33:05.603738+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:05.603738+0000","last_undegraded":"2024-04-22T08:33:05.603738+0000","last_fullsized":"2024-04-22T08:33:05.603738+0000","mapping_epoch":27,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":7246,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12748","reported_seq":13171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.677414+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:05.677414+0000","last_peered":"2024-04-22T08:33:05.677414+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:05.677414+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29143064,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9575,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":941},{"shards":"2","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13006","reported_seq":13039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.559159+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:05.559159+0000","last_peered":"2024-04-22T08:33:05.559159+0000","last_clean":"2024-04-22T08:33:05.559159+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:05.559159+0000","last_undegraded":"2024-04-22T08:33:05.559159+0000","last_fullsized":"2024-04-22T08:33:05.559159+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29114368,"num_objects":7108,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13006,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12835","reported_seq":12976,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.784790+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:05.784790+0000","last_peered":"2024-04-22T08:33:05.784790+0000","last_clean":"2024-04-22T08:33:05.784790+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:05.784790+0000","last_undegraded":"2024-04-22T08:33:05.784790+0000","last_fullsized":"2024-04-22T08:33:05.784790+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12960","reported_seq":15502,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.841503+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:05.841503+0000","last_peered":"2024-04-22T08:33:05.841503+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:05.841503+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1751,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":4026368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4394},{"shards":"0,2","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12760","reported_seq":20499,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:04.531757+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:04.531757+0000","last_peered":"2024-04-22T08:33:04.531757+0000","last_clean":"2024-04-22T08:33:04.531757+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:04.531757+0000","last_undegraded":"2024-04-22T08:33:04.531757+0000","last_fullsized":"2024-04-22T08:33:04.531757+0000","mapping_epoch":36,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":7104,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12741","reported_seq":12888,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.758386+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:05.758386+0000","last_peered":"2024-04-22T08:33:05.758386+0000","last_clean":"2024-04-22T08:33:05.758386+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:05.758386+0000","last_undegraded":"2024-04-22T08:33:05.758386+0000","last_fullsized":"2024-04-22T08:33:05.758386+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28938240,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13107","reported_seq":13493,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.684756+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:05.684756+0000","last_peered":"2024-04-22T08:33:05.684756+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:05.684756+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29140178,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":924},{"shards":"2","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12947","reported_seq":13635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.835994+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:05.835994+0000","last_peered":"2024-04-22T08:33:05.835994+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:05.835994+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652178,"num_objects":7241,"num_object_clones":0,"num_object_copies":14482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12945,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3774},{"shards":"0,2","objects":921}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12763","reported_seq":13226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.691306+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:05.691306+0000","last_peered":"2024-04-22T08:33:05.691306+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:05.691306+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29351936,"num_objects":7166,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":954},{"shards":"2","objects":3573}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12858","reported_seq":20536,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.716253+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:05.716253+0000","last_peered":"2024-04-22T08:33:05.716253+0000","last_clean":"2024-04-22T08:33:05.716253+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:05.716253+0000","last_undegraded":"2024-04-22T08:33:05.716253+0000","last_fullsized":"2024-04-22T08:33:05.716253+0000","mapping_epoch":38,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984530,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12863","reported_seq":13447,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.844974+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:05.844974+0000","last_peered":"2024-04-22T08:33:05.844974+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:05.844974+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29815401,"num_objects":7280,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12767","reported_seq":13305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.794525+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:05.794525+0000","last_peered":"2024-04-22T08:33:05.794525+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:05.794525+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28602368,"num_objects":6983,"num_object_clones":0,"num_object_copies":13966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3326},{"shards":"0,2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12849","reported_seq":13308,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.671965+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:05.671965+0000","last_peered":"2024-04-22T08:33:05.671965+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:05.671965+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447378,"num_objects":7191,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3676},{"shards":"0,2","objects":877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12969","reported_seq":13581,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.815241+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:05.815241+0000","last_peered":"2024-04-22T08:33:05.815241+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:05.815241+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":467414893,"num_objects":114127,"num_object_clones":0,"num_object_copies":228254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":114127,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179577,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1083420672,"data_stored":1083346427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160802,"ondisk_log_size":160802,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117069,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652876,"kb_used_data":350528,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93718964,"statfs":{"total":96636764160,"available":95968219136,"internally_reserved":0,"allocated":358940672,"data_stored":358714745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.217},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.245}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.883},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.32900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920831,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634432,"kb_used_data":305848,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93737408,"statfs":{"total":96636764160,"available":95987105792,"internally_reserved":0,"allocated":313188352,"data_stored":312906761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":6.9560000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313297,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756896,"kb_used_data":404568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93614944,"statfs":{"total":96636764160,"available":95861702656,"internally_reserved":0,"allocated":414277632,"data_stored":414062239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360771283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.8180000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":3.6829999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412827648,"data_stored":412824169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312819712,"data_stored":312756497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357773312,"data_stored":357765761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:09.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:10.273 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:10.311 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-04-22T08:33:08.295481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":467414893,"num_objects":114127,"num_object_clones":0,"num_object_copies":228254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":114127,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179577,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160802,"ondisk_log_size":160802,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2044204,"kb_used_data":1060944,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281071316,"statfs":{"total":289910292480,"available":287817027584,"internally_reserved":0,"allocated":1086406656,"data_stored":1085683745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1006816366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,7,3,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17211392,"num_objects":-4202,"num_object_clones":0,"num_object_copies":-8404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":377,"num_bytes_recovered":929792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007030"},"pg_stats":[{"pgid":"2.f","version":"39'12943","reported_seq":13513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.767937+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:05.767937+0000","last_peered":"2024-04-22T08:33:05.767937+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:05.767937+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29593600,"num_objects":7225,"num_object_clones":0,"num_object_copies":14450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12943,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3683},{"shards":"0,2","objects":935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12986","reported_seq":13104,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.603738+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:05.603738+0000","last_peered":"2024-04-22T08:33:05.603738+0000","last_clean":"2024-04-22T08:33:05.603738+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:05.603738+0000","last_undegraded":"2024-04-22T08:33:05.603738+0000","last_fullsized":"2024-04-22T08:33:05.603738+0000","mapping_epoch":27,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":7246,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12748","reported_seq":13171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.677414+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:05.677414+0000","last_peered":"2024-04-22T08:33:05.677414+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:05.677414+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29143064,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9575,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":941},{"shards":"2","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13006","reported_seq":13039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.559159+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:05.559159+0000","last_peered":"2024-04-22T08:33:05.559159+0000","last_clean":"2024-04-22T08:33:05.559159+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:05.559159+0000","last_undegraded":"2024-04-22T08:33:05.559159+0000","last_fullsized":"2024-04-22T08:33:05.559159+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29114368,"num_objects":7108,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13006,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12835","reported_seq":12976,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.784790+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:05.784790+0000","last_peered":"2024-04-22T08:33:05.784790+0000","last_clean":"2024-04-22T08:33:05.784790+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:05.784790+0000","last_undegraded":"2024-04-22T08:33:05.784790+0000","last_fullsized":"2024-04-22T08:33:05.784790+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12960","reported_seq":15502,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.841503+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:05.841503+0000","last_peered":"2024-04-22T08:33:05.841503+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:05.841503+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1751,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":4026368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4394},{"shards":"0,2","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12760","reported_seq":20499,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:04.531757+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:04.531757+0000","last_peered":"2024-04-22T08:33:04.531757+0000","last_clean":"2024-04-22T08:33:04.531757+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:04.531757+0000","last_undegraded":"2024-04-22T08:33:04.531757+0000","last_fullsized":"2024-04-22T08:33:04.531757+0000","mapping_epoch":36,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":7104,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12741","reported_seq":12888,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.758386+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:05.758386+0000","last_peered":"2024-04-22T08:33:05.758386+0000","last_clean":"2024-04-22T08:33:05.758386+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:05.758386+0000","last_undegraded":"2024-04-22T08:33:05.758386+0000","last_fullsized":"2024-04-22T08:33:05.758386+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28938240,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13107","reported_seq":13493,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.684756+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:05.684756+0000","last_peered":"2024-04-22T08:33:05.684756+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:05.684756+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29140178,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":924},{"shards":"2","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12947","reported_seq":13635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.835994+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:05.835994+0000","last_peered":"2024-04-22T08:33:05.835994+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:05.835994+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652178,"num_objects":7241,"num_object_clones":0,"num_object_copies":14482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12945,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3774},{"shards":"0,2","objects":921}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12763","reported_seq":13226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.691306+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:05.691306+0000","last_peered":"2024-04-22T08:33:05.691306+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:05.691306+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29351936,"num_objects":7166,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":954},{"shards":"2","objects":3573}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12858","reported_seq":20536,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.716253+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:05.716253+0000","last_peered":"2024-04-22T08:33:05.716253+0000","last_clean":"2024-04-22T08:33:05.716253+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:05.716253+0000","last_undegraded":"2024-04-22T08:33:05.716253+0000","last_fullsized":"2024-04-22T08:33:05.716253+0000","mapping_epoch":38,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984530,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12863","reported_seq":13447,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.844974+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:05.844974+0000","last_peered":"2024-04-22T08:33:05.844974+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:05.844974+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29815401,"num_objects":7280,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12767","reported_seq":13305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.794525+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:05.794525+0000","last_peered":"2024-04-22T08:33:05.794525+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:05.794525+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28602368,"num_objects":6983,"num_object_clones":0,"num_object_copies":13966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3326},{"shards":"0,2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12849","reported_seq":13308,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.671965+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:05.671965+0000","last_peered":"2024-04-22T08:33:05.671965+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:05.671965+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447378,"num_objects":7191,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3676},{"shards":"0,2","objects":877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12969","reported_seq":13581,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.815241+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:05.815241+0000","last_peered":"2024-04-22T08:33:05.815241+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:05.815241+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":467414893,"num_objects":114127,"num_object_clones":0,"num_object_copies":228254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":114127,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179577,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1083420672,"data_stored":1083346427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160802,"ondisk_log_size":160802,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117069,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652876,"kb_used_data":350528,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93718964,"statfs":{"total":96636764160,"available":95968219136,"internally_reserved":0,"allocated":358940672,"data_stored":358714745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.217},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.245}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.883},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.32900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920831,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634432,"kb_used_data":305848,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93737408,"statfs":{"total":96636764160,"available":95987105792,"internally_reserved":0,"allocated":313188352,"data_stored":312906761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":6.9560000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313297,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756896,"kb_used_data":404568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93614944,"statfs":{"total":96636764160,"available":95861702656,"internally_reserved":0,"allocated":414277632,"data_stored":414062239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360771283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.8180000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":3.6829999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412827648,"data_stored":412824169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312819712,"data_stored":312756497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357773312,"data_stored":357765761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:10.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:10.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'12943", 'reported_seq': 13513, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.767937+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:05.767937+0000', 'last_peered': '2024-04-22T08:33:05.767937+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:05.767937+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29593600, 'num_objects': 7225, 'num_object_clones': 0, 'num_object_copies': 14450, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3542, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12943, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 252, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3683}, {'shards': '0,2', 'objects': 935}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12748", 'reported_seq': 13171, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.677414+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:05.677414+0000', 'last_peered': '2024-04-22T08:33:05.677414+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:05.677414+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2700", 'ondisk_log_start': "28'2700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29143064, 'num_objects': 7116, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3608, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7116, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9575, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 941}, {'shards': '2', 'objects': 3509}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'12960", 'reported_seq': 15502, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.841503+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:05.841503+0000', 'last_peered': '2024-04-22T08:33:05.841503+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:05.841503+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29028352, 'num_objects': 7087, 'num_object_clones': 0, 'num_object_copies': 14174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1751, 'num_objects_misplaced': 942, 'num_objects_unfound': 0, 'num_objects_dirty': 7087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9829, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1189, 'num_bytes_recovered': 4026368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4394}, {'shards': '0,2', 'objects': 942}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13107", 'reported_seq': 13493, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.684756+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:05.684756+0000', 'last_peered': '2024-04-22T08:33:05.684756+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:05.684756+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'3100", 'ondisk_log_start': "28'3100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29140178, 'num_objects': 7116, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13047, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 135, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 924}, {'shards': '2', 'objects': 3358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'12947", 'reported_seq': 13635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.835994+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:05.835994+0000', 'last_peered': '2024-04-22T08:33:05.835994+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:05.835994+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29652178, 'num_objects': 7241, 'num_object_clones': 0, 'num_object_copies': 14482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3469, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12945, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 307, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3774}, {'shards': '0,2', 'objects': 921}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:10.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12763", 'reported_seq': 13226, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.691306+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:05.691306+0000', 'last_peered': '2024-04-22T08:33:05.691306+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:05.691306+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2700", 'ondisk_log_start': "28'2700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29351936, 'num_objects': 7166, 'num_object_clones': 0, 'num_object_copies': 14332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9217, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 157, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 954}, {'shards': '2', 'objects': 3573}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12863", 'reported_seq': 13447, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.844974+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:05.844974+0000', 'last_peered': '2024-04-22T08:33:05.844974+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:05.844974+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29815401, 'num_objects': 7280, 'num_object_clones': 0, 'num_object_copies': 14560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3471, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12862, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 253, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3810}, {'shards': '0,2', 'objects': 930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12767", 'reported_seq': 13305, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.794525+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:05.794525+0000', 'last_peered': '2024-04-22T08:33:05.794525+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:05.794525+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2700", 'ondisk_log_start': "28'2700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28602368, 'num_objects': 6983, 'num_object_clones': 0, 'num_object_copies': 13966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3660, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8998, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 180, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3326}, {'shards': '0,2', 'objects': 949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12849", 'reported_seq': 13308, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.671965+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:05.671965+0000', 'last_peered': '2024-04-22T08:33:05.671965+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:05.671965+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29447378, 'num_objects': 7191, 'num_object_clones': 0, 'num_object_copies': 14382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3517, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12847, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 192, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3676}, {'shards': '0,2', 'objects': 877}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:10.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'12969", 'reported_seq': 13581, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:05.815241+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:05.815241+0000', 'last_peered': '2024-04-22T08:33:05.815241+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:05.815241+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28809636, 'num_objects': 7037, 'num_object_clones': 0, 'num_object_copies': 14074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3651, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12966, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 238, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3389}, {'shards': '0,2', 'objects': 919}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:10.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:11.200 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:11.200 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34024/227958 objects degraded (14.926%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":426,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":113979,"data_bytes":466808685,"bytes_used":2083778560,"bytes_avail":287826513920,"bytes_total":289910292480,"degraded_objects":34024,"degraded_total":227958,"degraded_ratio":0.14925556462155309,"misplaced_objects":942,"misplaced_total":227958,"misplaced_ratio":0.0041323401679256707,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":77438,"recovering_keys_per_sec":0,"num_objects_recovered":377,"num_bytes_recovered":929792,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":362},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:11.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:11.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:11.990 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-04-22T08:33:10.295918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466808685,"num_objects":113979,"num_object_clones":0,"num_object_copies":227958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":113979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179725,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2034940,"kb_used_data":1059872,"kb_used_omap":12,"kb_used_meta":975027,"kb_avail":281080580,"statfs":{"total":289910292480,"available":287826513920,"internally_reserved":0,"allocated":1085308928,"data_stored":1084586017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":998427758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,7,3,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17817600,"num_objects":-4350,"num_object_clones":0,"num_object_copies":-8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":377,"num_bytes_recovered":929792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006919"},"pg_stats":[{"pgid":"2.f","version":"39'12943","reported_seq":13513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.767937+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:05.767937+0000","last_peered":"2024-04-22T08:33:05.767937+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:05.767937+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29593600,"num_objects":7225,"num_object_clones":0,"num_object_copies":14450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12943,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3683},{"shards":"0,2","objects":935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'12986","reported_seq":13104,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.603738+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:05.603738+0000","last_peered":"2024-04-22T08:33:05.603738+0000","last_clean":"2024-04-22T08:33:05.603738+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:05.603738+0000","last_undegraded":"2024-04-22T08:33:05.603738+0000","last_fullsized":"2024-04-22T08:33:05.603738+0000","mapping_epoch":27,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":7246,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12748","reported_seq":13171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.677414+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:05.677414+0000","last_peered":"2024-04-22T08:33:05.677414+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:05.677414+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29143064,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9575,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":941},{"shards":"2","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13006","reported_seq":13039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.559159+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:05.559159+0000","last_peered":"2024-04-22T08:33:05.559159+0000","last_clean":"2024-04-22T08:33:05.559159+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:05.559159+0000","last_undegraded":"2024-04-22T08:33:05.559159+0000","last_fullsized":"2024-04-22T08:33:05.559159+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29114368,"num_objects":7108,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13006,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12835","reported_seq":12976,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.784790+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:05.784790+0000","last_peered":"2024-04-22T08:33:05.784790+0000","last_clean":"2024-04-22T08:33:05.784790+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:05.784790+0000","last_undegraded":"2024-04-22T08:33:05.784790+0000","last_fullsized":"2024-04-22T08:33:05.784790+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'12960","reported_seq":15502,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.841503+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:05.841503+0000","last_peered":"2024-04-22T08:33:05.841503+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:05.841503+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1751,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":4026368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4394},{"shards":"0,2","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12908","reported_seq":20647,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:09.655009+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:09.655009+0000","last_peered":"2024-04-22T08:33:09.655009+0000","last_clean":"2024-04-22T08:33:09.655009+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:09.655009+0000","last_undegraded":"2024-04-22T08:33:09.655009+0000","last_fullsized":"2024-04-22T08:33:09.655009+0000","mapping_epoch":36,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":6956,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12741","reported_seq":12888,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.758386+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:05.758386+0000","last_peered":"2024-04-22T08:33:05.758386+0000","last_clean":"2024-04-22T08:33:05.758386+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:05.758386+0000","last_undegraded":"2024-04-22T08:33:05.758386+0000","last_fullsized":"2024-04-22T08:33:05.758386+0000","mapping_epoch":27,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28938240,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13107","reported_seq":13493,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.684756+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:05.684756+0000","last_peered":"2024-04-22T08:33:05.684756+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:05.684756+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29140178,"num_objects":7116,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":924},{"shards":"2","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'12947","reported_seq":13635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.835994+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:05.835994+0000","last_peered":"2024-04-22T08:33:05.835994+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:05.835994+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652178,"num_objects":7241,"num_object_clones":0,"num_object_copies":14482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12945,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3774},{"shards":"0,2","objects":921}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12763","reported_seq":13226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.691306+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:05.691306+0000","last_peered":"2024-04-22T08:33:05.691306+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:05.691306+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29351936,"num_objects":7166,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":954},{"shards":"2","objects":3573}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12858","reported_seq":20536,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.716253+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:05.716253+0000","last_peered":"2024-04-22T08:33:05.716253+0000","last_clean":"2024-04-22T08:33:05.716253+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:05.716253+0000","last_undegraded":"2024-04-22T08:33:05.716253+0000","last_fullsized":"2024-04-22T08:33:05.716253+0000","mapping_epoch":38,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984530,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12863","reported_seq":13447,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.844974+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:05.844974+0000","last_peered":"2024-04-22T08:33:05.844974+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:05.844974+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29815401,"num_objects":7280,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12862,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12767","reported_seq":13305,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.794525+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:05.794525+0000","last_peered":"2024-04-22T08:33:05.794525+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:05.794525+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2700","ondisk_log_start":"28'2700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28602368,"num_objects":6983,"num_object_clones":0,"num_object_copies":13966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3326},{"shards":"0,2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12849","reported_seq":13308,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.671965+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:05.671965+0000","last_peered":"2024-04-22T08:33:05.671965+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:05.671965+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447378,"num_objects":7191,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3676},{"shards":"0,2","objects":877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'12969","reported_seq":13581,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:05.815241+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:05.815241+0000","last_peered":"2024-04-22T08:33:05.815241+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:05.815241+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":466808685,"num_objects":113979,"num_object_clones":0,"num_object_copies":227958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34024,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":113979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179725,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10641,"num_bytes_recovered":35103337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1082265600,"data_stored":1082191355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117069,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652876,"kb_used_data":350528,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93718964,"statfs":{"total":96636764160,"available":95968219136,"internally_reserved":0,"allocated":358940672,"data_stored":358714745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.217},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":0.245}]},{"osd":1,"last update":"Mon Apr 22 08:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1279999999999999,"15min":1},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.247999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.883},{"interface":"front","average":{"1min":2.3260000000000001,"5min":1.075,"15min":0.74099999999999999},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":7.4640000000000004},"last":0.32900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920832,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625168,"kb_used_data":304776,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93746672,"statfs":{"total":96636764160,"available":95996592128,"internally_reserved":0,"allocated":312090624,"data_stored":311809033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":328068812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":6.9560000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313297,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756896,"kb_used_data":404568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93614944,"statfs":{"total":96636764160,"available":95861702656,"internally_reserved":0,"allocated":414277632,"data_stored":414062239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":360771283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":2.8180000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":3.6829999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412827648,"data_stored":412824169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311664640,"data_stored":311601425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357773312,"data_stored":357765761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:14.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:15.444 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:15.445 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:15.461 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-04-22T08:33:14.296896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458690413,"num_objects":111997,"num_object_clones":0,"num_object_copies":223994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33949,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":111997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181707,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10854,"num_bytes_recovered":35603049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2049864,"kb_used_data":1048556,"kb_used_omap":12,"kb_used_meta":1001267,"kb_avail":281065656,"statfs":{"total":289910292480,"available":287811231744,"internally_reserved":0,"allocated":1073721344,"data_stored":1072998433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1025297518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,6,4,3,4,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17371136,"num_objects":-4241,"num_object_clones":0,"num_object_copies":-8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":408,"num_bytes_recovered":970752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005417"},"pg_stats":[{"pgid":"2.f","version":"39'13072","reported_seq":13660,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.818020+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:10.818020+0000","last_peered":"2024-04-22T08:33:10.818020+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:10.818020+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29065216,"num_objects":7096,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13072,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3564},{"shards":"0,2","objects":806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13122","reported_seq":13240,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.714844+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:10.714844+0000","last_peered":"2024-04-22T08:33:10.714844+0000","last_clean":"2024-04-22T08:33:10.714844+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:10.714844+0000","last_undegraded":"2024-04-22T08:33:10.714844+0000","last_fullsized":"2024-04-22T08:33:10.714844+0000","mapping_epoch":27,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":7110,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12900","reported_seq":13345,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.699323+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:10.699323+0000","last_peered":"2024-04-22T08:33:10.699323+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:10.699323+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28520472,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9727,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":789},{"shards":"2","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13126","reported_seq":13159,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.663672+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:10.663672+0000","last_peered":"2024-04-22T08:33:10.663672+0000","last_clean":"2024-04-22T08:33:10.663672+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:10.663672+0000","last_undegraded":"2024-04-22T08:33:10.663672+0000","last_fullsized":"2024-04-22T08:33:10.663672+0000","mapping_epoch":27,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28622848,"num_objects":6988,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13126,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12955","reported_seq":13096,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.841286+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:10.841286+0000","last_peered":"2024-04-22T08:33:10.841286+0000","last_clean":"2024-04-22T08:33:10.841286+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:10.841286+0000","last_undegraded":"2024-04-22T08:33:10.841286+0000","last_fullsized":"2024-04-22T08:33:10.841286+0000","mapping_epoch":27,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13094","reported_seq":15884,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.873357+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:10.873357+0000","last_peered":"2024-04-22T08:33:10.873357+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:10.873357+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1765,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9963,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":4526080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4384},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12908","reported_seq":20647,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:09.655009+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:09.655009+0000","last_peered":"2024-04-22T08:33:09.655009+0000","last_clean":"2024-04-22T08:33:09.655009+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:09.655009+0000","last_undegraded":"2024-04-22T08:33:09.655009+0000","last_fullsized":"2024-04-22T08:33:09.655009+0000","mapping_epoch":36,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":6956,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12893","reported_seq":13040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.810310+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:10.810310+0000","last_peered":"2024-04-22T08:33:10.810310+0000","last_clean":"2024-04-22T08:33:10.810310+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:10.810310+0000","last_undegraded":"2024-04-22T08:33:10.810310+0000","last_fullsized":"2024-04-22T08:33:10.810310+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28315648,"num_objects":6913,"num_object_clones":0,"num_object_copies":13826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13211","reported_seq":13603,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.585123+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:10.585123+0000","last_peered":"2024-04-22T08:33:10.585123+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:10.585123+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28714194,"num_objects":7012,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13151,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":820},{"shards":"2","objects":3257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13082","reported_seq":13793,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.774505+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:10.774505+0000","last_peered":"2024-04-22T08:33:10.774505+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:10.774505+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29099218,"num_objects":7106,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13080,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3650},{"shards":"0,2","objects":786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12901","reported_seq":13380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.733764+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:10.733764+0000","last_peered":"2024-04-22T08:33:10.733764+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:10.733764+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28786688,"num_objects":7028,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":816},{"shards":"2","objects":3444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12976","reported_seq":20654,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.565917+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:10.565917+0000","last_peered":"2024-04-22T08:33:10.565917+0000","last_clean":"2024-04-22T08:33:10.565917+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:10.565917+0000","last_undegraded":"2024-04-22T08:33:10.565917+0000","last_fullsized":"2024-04-22T08:33:10.565917+0000","mapping_epoch":38,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28501202,"num_objects":6960,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12974,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12999","reported_seq":13601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.644954+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:10.644954+0000","last_peered":"2024-04-22T08:33:10.644954+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:10.644954+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29258345,"num_objects":7144,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12998,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3683},{"shards":"0,2","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12903","reported_seq":13475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.795543+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:10.795543+0000","last_peered":"2024-04-22T08:33:10.795543+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:10.795543+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28045312,"num_objects":6847,"num_object_clones":0,"num_object_copies":13694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3207},{"shards":"0,2","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12984","reported_seq":13475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.880615+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:10.880615+0000","last_peered":"2024-04-22T08:33:10.880615+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:10.880615+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28894418,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12982,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3557},{"shards":"0,2","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13106","reported_seq":13725,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.866428+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:10.866428+0000","last_peered":"2024-04-22T08:33:10.866428+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:10.866428+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28248484,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13103,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3255},{"shards":"0,2","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":458690413,"num_objects":111997,"num_object_clones":0,"num_object_copies":223994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33949,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":111997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181707,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10854,"num_bytes_recovered":35603049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1071394816,"data_stored":1071320571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117070,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652004,"kb_used_data":345560,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93719836,"statfs":{"total":96636764160,"available":95969112064,"internally_reserved":0,"allocated":353853440,"data_stored":353627513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":6.7670000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":10.699},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.29399999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920832,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625168,"kb_used_data":304776,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93746672,"statfs":{"total":96636764160,"available":95996592128,"internally_reserved":0,"allocated":312090624,"data_stored":311809033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":328068812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":6.9560000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772692,"kb_used_data":398220,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93599148,"statfs":{"total":96636764160,"available":95845527552,"internally_reserved":0,"allocated":407777280,"data_stored":407561887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,0,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":406216704,"data_stored":406213225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311664640,"data_stored":311601425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353513472,"data_stored":353505921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:15.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:16.188 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:16.189 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:16.233 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-04-22T08:33:14.296896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458690413,"num_objects":111997,"num_object_clones":0,"num_object_copies":223994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33949,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":111997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181707,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10854,"num_bytes_recovered":35603049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2049864,"kb_used_data":1048556,"kb_used_omap":12,"kb_used_meta":1001267,"kb_avail":281065656,"statfs":{"total":289910292480,"available":287811231744,"internally_reserved":0,"allocated":1073721344,"data_stored":1072998433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1025297518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,6,4,3,4,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17371136,"num_objects":-4241,"num_object_clones":0,"num_object_copies":-8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":408,"num_bytes_recovered":970752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005417"},"pg_stats":[{"pgid":"2.f","version":"39'13072","reported_seq":13660,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.818020+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:10.818020+0000","last_peered":"2024-04-22T08:33:10.818020+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:10.818020+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29065216,"num_objects":7096,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13072,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3564},{"shards":"0,2","objects":806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13122","reported_seq":13240,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.714844+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:10.714844+0000","last_peered":"2024-04-22T08:33:10.714844+0000","last_clean":"2024-04-22T08:33:10.714844+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:10.714844+0000","last_undegraded":"2024-04-22T08:33:10.714844+0000","last_fullsized":"2024-04-22T08:33:10.714844+0000","mapping_epoch":27,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":7110,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'12900","reported_seq":13345,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.699323+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:10.699323+0000","last_peered":"2024-04-22T08:33:10.699323+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:10.699323+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28520472,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9727,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":789},{"shards":"2","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13126","reported_seq":13159,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.663672+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:10.663672+0000","last_peered":"2024-04-22T08:33:10.663672+0000","last_clean":"2024-04-22T08:33:10.663672+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:10.663672+0000","last_undegraded":"2024-04-22T08:33:10.663672+0000","last_fullsized":"2024-04-22T08:33:10.663672+0000","mapping_epoch":27,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28622848,"num_objects":6988,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13126,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'12955","reported_seq":13096,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.841286+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:10.841286+0000","last_peered":"2024-04-22T08:33:10.841286+0000","last_clean":"2024-04-22T08:33:10.841286+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:10.841286+0000","last_undegraded":"2024-04-22T08:33:10.841286+0000","last_fullsized":"2024-04-22T08:33:10.841286+0000","mapping_epoch":27,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13094","reported_seq":15884,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.873357+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:10.873357+0000","last_peered":"2024-04-22T08:33:10.873357+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:10.873357+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1765,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9963,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":4526080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4384},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'12908","reported_seq":20647,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:09.655009+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:09.655009+0000","last_peered":"2024-04-22T08:33:09.655009+0000","last_clean":"2024-04-22T08:33:09.655009+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:09.655009+0000","last_undegraded":"2024-04-22T08:33:09.655009+0000","last_fullsized":"2024-04-22T08:33:09.655009+0000","mapping_epoch":36,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":6956,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'12893","reported_seq":13040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.810310+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:10.810310+0000","last_peered":"2024-04-22T08:33:10.810310+0000","last_clean":"2024-04-22T08:33:10.810310+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:10.810310+0000","last_undegraded":"2024-04-22T08:33:10.810310+0000","last_fullsized":"2024-04-22T08:33:10.810310+0000","mapping_epoch":27,"log_start":"28'2800","ondisk_log_start":"28'2800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28315648,"num_objects":6913,"num_object_clones":0,"num_object_copies":13826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13211","reported_seq":13603,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.585123+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:10.585123+0000","last_peered":"2024-04-22T08:33:10.585123+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:10.585123+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28714194,"num_objects":7012,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13151,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":820},{"shards":"2","objects":3257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13082","reported_seq":13793,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.774505+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:10.774505+0000","last_peered":"2024-04-22T08:33:10.774505+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:10.774505+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29099218,"num_objects":7106,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13080,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3650},{"shards":"0,2","objects":786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'12901","reported_seq":13380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.733764+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:10.733764+0000","last_peered":"2024-04-22T08:33:10.733764+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:10.733764+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28786688,"num_objects":7028,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":816},{"shards":"2","objects":3444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'12976","reported_seq":20654,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:10.565917+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:10.565917+0000","last_peered":"2024-04-22T08:33:10.565917+0000","last_clean":"2024-04-22T08:33:10.565917+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:10.565917+0000","last_undegraded":"2024-04-22T08:33:10.565917+0000","last_fullsized":"2024-04-22T08:33:10.565917+0000","mapping_epoch":38,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28501202,"num_objects":6960,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12974,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'12999","reported_seq":13601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.644954+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:10.644954+0000","last_peered":"2024-04-22T08:33:10.644954+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:10.644954+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29258345,"num_objects":7144,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12998,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3683},{"shards":"0,2","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'12903","reported_seq":13475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.795543+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:10.795543+0000","last_peered":"2024-04-22T08:33:10.795543+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:10.795543+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28045312,"num_objects":6847,"num_object_clones":0,"num_object_copies":13694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3207},{"shards":"0,2","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'12984","reported_seq":13475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.880615+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:10.880615+0000","last_peered":"2024-04-22T08:33:10.880615+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:10.880615+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'2900","ondisk_log_start":"28'2900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28894418,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12982,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3557},{"shards":"0,2","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13106","reported_seq":13725,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:10.866428+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:10.866428+0000","last_peered":"2024-04-22T08:33:10.866428+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:10.866428+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28248484,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13103,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3255},{"shards":"0,2","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":458690413,"num_objects":111997,"num_object_clones":0,"num_object_copies":223994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33949,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":111997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181707,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10854,"num_bytes_recovered":35603049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1071394816,"data_stored":1071320571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117070,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652004,"kb_used_data":345560,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93719836,"statfs":{"total":96636764160,"available":95969112064,"internally_reserved":0,"allocated":353853440,"data_stored":353627513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92500000000000004},"min":{"1min":0.22800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":6.673,"5min":13.635,"15min":13.635},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.0229999999999999,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.044,"5min":13.721,"15min":13.721},"last":6.7670000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":10.699},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.29399999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920832,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625168,"kb_used_data":304776,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93746672,"statfs":{"total":96636764160,"available":95996592128,"internally_reserved":0,"allocated":312090624,"data_stored":311809033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":328068812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":6.9560000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.9299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772692,"kb_used_data":398220,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93599148,"statfs":{"total":96636764160,"available":95845527552,"internally_reserved":0,"allocated":407777280,"data_stored":407561887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,3,0,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.0820000000000001,"15min":0.82099999999999995},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":8,"5min":8,"15min":8},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.85399999999999998,"15min":0.80400000000000005},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.5190000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":406216704,"data_stored":406213225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311664640,"data_stored":311601425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353513472,"data_stored":353505921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:16.234 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:16.234 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'13072", 'reported_seq': 13660, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.818020+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:10.818020+0000', 'last_peered': '2024-04-22T08:33:10.818020+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:10.818020+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29065216, 'num_objects': 7096, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3532, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13072, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 262, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3564}, {'shards': '0,2', 'objects': 806}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.234 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:16.234 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'12900", 'reported_seq': 13345, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.699323+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:10.699323+0000', 'last_peered': '2024-04-22T08:33:10.699323+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:10.699323+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'2800", 'ondisk_log_start': "28'2800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28520472, 'num_objects': 6964, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3597, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6964, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9727, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 159, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 789}, {'shards': '2', 'objects': 3368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'13094", 'reported_seq': 15884, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.873357+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:10.873357+0000', 'last_peered': '2024-04-22T08:33:10.873357+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:10.873357+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28479488, 'num_objects': 6953, 'num_object_clones': 0, 'num_object_copies': 13906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1765, 'num_objects_misplaced': 804, 'num_objects_unfound': 0, 'num_objects_dirty': 6953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9963, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1313, 'num_bytes_recovered': 4526080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4384}, {'shards': '0,2', 'objects': 804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13211", 'reported_seq': 13603, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.585123+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:10.585123+0000', 'last_peered': '2024-04-22T08:33:10.585123+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:10.585123+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28714194, 'num_objects': 7012, 'num_object_clones': 0, 'num_object_copies': 14024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3757, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13151, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 138, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 820}, {'shards': '2', 'objects': 3257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'13082", 'reported_seq': 13793, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.774505+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:10.774505+0000', 'last_peered': '2024-04-22T08:33:10.774505+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:10.774505+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29099218, 'num_objects': 7106, 'num_object_clones': 0, 'num_object_copies': 14212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3458, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13080, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 318, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3650}, {'shards': '0,2', 'objects': 786}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'12901", 'reported_seq': 13380, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.733764+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:10.733764+0000', 'last_peered': '2024-04-22T08:33:10.733764+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:10.733764+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28786688, 'num_objects': 7028, 'num_object_clones': 0, 'num_object_copies': 14056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3586, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9355, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 166, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 816}, {'shards': '2', 'objects': 3444}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'12999", 'reported_seq': 13601, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.644954+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:10.644954+0000', 'last_peered': '2024-04-22T08:33:10.644954+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:10.644954+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29258345, 'num_objects': 7144, 'num_object_clones': 0, 'num_object_copies': 14288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3462, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12998, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 262, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3683}, {'shards': '0,2', 'objects': 794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.235 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:16.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'12903", 'reported_seq': 13475, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.795543+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:10.795543+0000', 'last_peered': '2024-04-22T08:33:10.795543+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:10.795543+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28045312, 'num_objects': 6847, 'num_object_clones': 0, 'num_object_copies': 13694, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3643, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9134, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 197, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3207}, {'shards': '0,2', 'objects': 813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:16.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'12984", 'reported_seq': 13475, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.880615+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:10.880615+0000', 'last_peered': '2024-04-22T08:33:10.880615+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:10.880615+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'2900", 'ondisk_log_start': "28'2900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28894418, 'num_objects': 7056, 'num_object_clones': 0, 'num_object_copies': 14112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3501, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12982, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 208, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3557}, {'shards': '0,2', 'objects': 742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:16.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'13106", 'reported_seq': 13725, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:10.866428+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:10.866428+0000', 'last_peered': '2024-04-22T08:33:10.866428+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:10.866428+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'3100", 'ondisk_log_start': "28'3100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28248484, 'num_objects': 6900, 'num_object_clones': 0, 'num_object_copies': 13800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3648, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13103, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 241, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3255}, {'shards': '0,2', 'objects': 782}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:16.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:16.952 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33897/219842 objects degraded (15.419%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":431,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":109921,"data_bytes":450187117,"bytes_used":2108928000,"bytes_avail":287801364480,"bytes_total":289910292480,"degraded_objects":33897,"degraded_total":219842,"degraded_ratio":0.15418800775102118,"misplaced_objects":675,"misplaced_total":219842,"misplaced_ratio":0.0030703869142384077,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":118048,"recovering_keys_per_sec":0,"num_objects_recovered":589,"num_bytes_recovered":1417216,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":526},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:17.000 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:17.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:17.839 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:17.891 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-04-22T08:33:16.297364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":450187117,"num_objects":109921,"num_object_clones":0,"num_object_copies":219842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2059500,"kb_used_data":1037712,"kb_used_omap":12,"kb_used_meta":1021747,"kb_avail":281056020,"statfs":{"total":289910292480,"available":287801364480,"internally_reserved":0,"allocated":1062617088,"data_stored":1061894177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1046269038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,4,6,9,4,0,0,3,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25874432,"num_objects":-6317,"num_object_clones":0,"num_object_copies":-12634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6317,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005342"},"pg_stats":[{"pgid":"2.f","version":"39'13209","reported_seq":13817,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.712687+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:15.712687+0000","last_peered":"2024-04-22T08:33:15.712687+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:15.712687+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28504064,"num_objects":6959,"num_object_clones":0,"num_object_copies":13918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13209,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3436},{"shards":"0,2","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13237","reported_seq":13355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.743607+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:15.743607+0000","last_peered":"2024-04-22T08:33:15.743607+0000","last_clean":"2024-04-22T08:33:15.743607+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:15.743607+0000","last_undegraded":"2024-04-22T08:33:15.743607+0000","last_fullsized":"2024-04-22T08:33:15.743607+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28651520,"num_objects":6995,"num_object_clones":0,"num_object_copies":13990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13026","reported_seq":13471,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.719635+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:15.719635+0000","last_peered":"2024-04-22T08:33:15.719635+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:15.719635+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28004376,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9853,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":663},{"shards":"2","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13253","reported_seq":13286,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.700350+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:15.700350+0000","last_peered":"2024-04-22T08:33:15.700350+0000","last_clean":"2024-04-22T08:33:15.700350+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:15.700350+0000","last_undegraded":"2024-04-22T08:33:15.700350+0000","last_fullsized":"2024-04-22T08:33:15.700350+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28102656,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13253,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13080","reported_seq":13221,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.830481+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:15.830481+0000","last_peered":"2024-04-22T08:33:15.830481+0000","last_clean":"2024-04-22T08:33:15.830481+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:15.830481+0000","last_undegraded":"2024-04-22T08:33:15.830481+0000","last_fullsized":"2024-04-22T08:33:15.830481+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10172,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13230","reported_seq":16244,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.883714+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:15.883714+0000","last_peered":"2024-04-22T08:33:15.883714+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:15.883714+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27922432,"num_objects":6817,"num_object_clones":0,"num_object_copies":13634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1782,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":6817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":4972544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13045","reported_seq":20784,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:14.570084+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:14.570084+0000","last_peered":"2024-04-22T08:33:14.570084+0000","last_clean":"2024-04-22T08:33:14.570084+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:14.570084+0000","last_undegraded":"2024-04-22T08:33:14.570084+0000","last_fullsized":"2024-04-22T08:33:14.570084+0000","mapping_epoch":36,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27930624,"num_objects":6819,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13013","reported_seq":13160,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.777808+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:15.777808+0000","last_peered":"2024-04-22T08:33:15.777808+0000","last_clean":"2024-04-22T08:33:15.777808+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:15.777808+0000","last_undegraded":"2024-04-22T08:33:15.777808+0000","last_fullsized":"2024-04-22T08:33:15.777808+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13348","reported_seq":13752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.750610+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:15.750610+0000","last_peered":"2024-04-22T08:33:15.750610+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:15.750610+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28153042,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":683},{"shards":"2","objects":3126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13217","reported_seq":13954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.848338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:15.848338+0000","last_peered":"2024-04-22T08:33:15.848338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:15.848338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546258,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13215,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3528},{"shards":"0,2","objects":651}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13046","reported_seq":13529,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.602171+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:15.602171+0000","last_peered":"2024-04-22T08:33:15.602171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:15.602171+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28192768,"num_objects":6883,"num_object_clones":0,"num_object_copies":13766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":671},{"shards":"2","objects":3300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13115","reported_seq":20793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.625198+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:15.625198+0000","last_peered":"2024-04-22T08:33:15.625198+0000","last_clean":"2024-04-22T08:33:15.625198+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:15.625198+0000","last_undegraded":"2024-04-22T08:33:15.625198+0000","last_fullsized":"2024-04-22T08:33:15.625198+0000","mapping_epoch":38,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27931858,"num_objects":6821,"num_object_clones":0,"num_object_copies":13642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13122","reported_seq":13756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.831939+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:15.831939+0000","last_peered":"2024-04-22T08:33:15.831939+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:15.831939+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28754537,"num_objects":7021,"num_object_clones":0,"num_object_copies":14042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13121,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3576},{"shards":"0,2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13031","reported_seq":13617,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.881591+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:15.881591+0000","last_peered":"2024-04-22T08:33:15.881591+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:15.881591+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27521024,"num_objects":6719,"num_object_clones":0,"num_object_copies":13438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3086},{"shards":"0,2","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13109","reported_seq":13616,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.877373+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:15.877373+0000","last_peered":"2024-04-22T08:33:15.877373+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:15.877373+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13107,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3440},{"shards":"0,2","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13227","reported_seq":13860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.842629+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:15.842629+0000","last_peered":"2024-04-22T08:33:15.842629+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:15.842629+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27752868,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":450187117,"num_objects":109921,"num_object_clones":0,"num_object_copies":219842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183783,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1059811328,"data_stored":1059737083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117071,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652728,"kb_used_data":342192,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93719112,"statfs":{"total":96636764160,"available":95968370688,"internally_reserved":0,"allocated":350404608,"data_stored":350178681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,6,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":7.556}]}]},{"osd":1,"up_from":33,"seq":141733920833,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632304,"kb_used_data":303716,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93739536,"statfs":{"total":96636764160,"available":95989284864,"internally_reserved":0,"allocated":311005184,"data_stored":310723593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":336457420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.50600000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313299,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774468,"kb_used_data":391804,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93597372,"statfs":{"total":96636764160,"available":95843708928,"internally_reserved":0,"allocated":401207296,"data_stored":400991903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,2,4,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399941632,"data_stored":399938153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310595584,"data_stored":310532369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349274112,"data_stored":349266561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:20.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:21.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:21.421 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:21.473 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-04-22T08:33:20.299935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449675117,"num_objects":109796,"num_object_clones":0,"num_object_copies":219592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183908,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160533,"ondisk_log_size":160533,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2064320,"kb_used_data":1036772,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281051200,"statfs":{"total":289910292480,"available":287796428800,"internally_reserved":0,"allocated":1061654528,"data_stored":1060931617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1052167278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,5,12,8,0,0,3,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17739776,"num_objects":-4331,"num_object_clones":0,"num_object_copies":-8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":946176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004453"},"pg_stats":[{"pgid":"2.f","version":"39'13209","reported_seq":13817,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.712687+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:15.712687+0000","last_peered":"2024-04-22T08:33:15.712687+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:15.712687+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28504064,"num_objects":6959,"num_object_clones":0,"num_object_copies":13918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13209,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3436},{"shards":"0,2","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13237","reported_seq":13355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.743607+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:15.743607+0000","last_peered":"2024-04-22T08:33:15.743607+0000","last_clean":"2024-04-22T08:33:15.743607+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:15.743607+0000","last_undegraded":"2024-04-22T08:33:15.743607+0000","last_fullsized":"2024-04-22T08:33:15.743607+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28651520,"num_objects":6995,"num_object_clones":0,"num_object_copies":13990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13026","reported_seq":13471,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.719635+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:15.719635+0000","last_peered":"2024-04-22T08:33:15.719635+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:15.719635+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28004376,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9853,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":663},{"shards":"2","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13253","reported_seq":13286,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.700350+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:15.700350+0000","last_peered":"2024-04-22T08:33:15.700350+0000","last_clean":"2024-04-22T08:33:15.700350+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:15.700350+0000","last_undegraded":"2024-04-22T08:33:15.700350+0000","last_fullsized":"2024-04-22T08:33:15.700350+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28102656,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13253,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13080","reported_seq":13221,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.830481+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:15.830481+0000","last_peered":"2024-04-22T08:33:15.830481+0000","last_clean":"2024-04-22T08:33:15.830481+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:15.830481+0000","last_undegraded":"2024-04-22T08:33:15.830481+0000","last_fullsized":"2024-04-22T08:33:15.830481+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10172,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13230","reported_seq":16244,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.883714+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:15.883714+0000","last_peered":"2024-04-22T08:33:15.883714+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:15.883714+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27922432,"num_objects":6817,"num_object_clones":0,"num_object_copies":13634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1782,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":6817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":4972544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13170","reported_seq":20909,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:19.683923+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:19.683923+0000","last_peered":"2024-04-22T08:33:19.683923+0000","last_clean":"2024-04-22T08:33:19.683923+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:19.683923+0000","last_undegraded":"2024-04-22T08:33:19.683923+0000","last_fullsized":"2024-04-22T08:33:19.683923+0000","mapping_epoch":36,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27418624,"num_objects":6694,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13013","reported_seq":13160,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.777808+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:15.777808+0000","last_peered":"2024-04-22T08:33:15.777808+0000","last_clean":"2024-04-22T08:33:15.777808+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:15.777808+0000","last_undegraded":"2024-04-22T08:33:15.777808+0000","last_fullsized":"2024-04-22T08:33:15.777808+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13348","reported_seq":13752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.750610+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:15.750610+0000","last_peered":"2024-04-22T08:33:15.750610+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:15.750610+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28153042,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":683},{"shards":"2","objects":3126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13217","reported_seq":13954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.848338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:15.848338+0000","last_peered":"2024-04-22T08:33:15.848338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:15.848338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546258,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13215,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3528},{"shards":"0,2","objects":651}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13046","reported_seq":13529,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.602171+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:15.602171+0000","last_peered":"2024-04-22T08:33:15.602171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:15.602171+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28192768,"num_objects":6883,"num_object_clones":0,"num_object_copies":13766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":671},{"shards":"2","objects":3300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13115","reported_seq":20793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.625198+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:15.625198+0000","last_peered":"2024-04-22T08:33:15.625198+0000","last_clean":"2024-04-22T08:33:15.625198+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:15.625198+0000","last_undegraded":"2024-04-22T08:33:15.625198+0000","last_fullsized":"2024-04-22T08:33:15.625198+0000","mapping_epoch":38,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27931858,"num_objects":6821,"num_object_clones":0,"num_object_copies":13642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13122","reported_seq":13756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.831939+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:15.831939+0000","last_peered":"2024-04-22T08:33:15.831939+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:15.831939+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28754537,"num_objects":7021,"num_object_clones":0,"num_object_copies":14042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13121,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3576},{"shards":"0,2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13031","reported_seq":13617,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.881591+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:15.881591+0000","last_peered":"2024-04-22T08:33:15.881591+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:15.881591+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27521024,"num_objects":6719,"num_object_clones":0,"num_object_copies":13438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3086},{"shards":"0,2","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13109","reported_seq":13616,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.877373+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:15.877373+0000","last_peered":"2024-04-22T08:33:15.877373+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:15.877373+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13107,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3440},{"shards":"0,2","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13227","reported_seq":13860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.842629+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:15.842629+0000","last_peered":"2024-04-22T08:33:15.842629+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:15.842629+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27752868,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":449675117,"num_objects":109796,"num_object_clones":0,"num_object_copies":219592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183908,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1058848768,"data_stored":1058774523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160533,"ondisk_log_size":160533,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117071,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652728,"kb_used_data":342192,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93719112,"statfs":{"total":96636764160,"available":95968370688,"internally_reserved":0,"allocated":350404608,"data_stored":350178681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,6,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":7.556}]}]},{"osd":1,"up_from":33,"seq":141733920834,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637124,"kb_used_data":302776,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93734716,"statfs":{"total":96636764160,"available":95984349184,"internally_reserved":0,"allocated":310042624,"data_stored":309761033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":342355660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.379}]}]},{"osd":0,"up_from":21,"seq":90194313299,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774468,"kb_used_data":391804,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93597372,"statfs":{"total":96636764160,"available":95843708928,"internally_reserved":0,"allocated":401207296,"data_stored":400991903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,2,4,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399941632,"data_stored":399938153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309633024,"data_stored":309569809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349274112,"data_stored":349266561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:21.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:22.137 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:22.137 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:22.218 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-04-22T08:33:20.299935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449675117,"num_objects":109796,"num_object_clones":0,"num_object_copies":219592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183908,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160533,"ondisk_log_size":160533,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2064320,"kb_used_data":1036772,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281051200,"statfs":{"total":289910292480,"available":287796428800,"internally_reserved":0,"allocated":1061654528,"data_stored":1060931617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1052167278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,5,12,8,0,0,3,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17739776,"num_objects":-4331,"num_object_clones":0,"num_object_copies":-8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":946176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004453"},"pg_stats":[{"pgid":"2.f","version":"39'13209","reported_seq":13817,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.712687+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:15.712687+0000","last_peered":"2024-04-22T08:33:15.712687+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:15.712687+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28504064,"num_objects":6959,"num_object_clones":0,"num_object_copies":13918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13209,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3436},{"shards":"0,2","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13237","reported_seq":13355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.743607+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:15.743607+0000","last_peered":"2024-04-22T08:33:15.743607+0000","last_clean":"2024-04-22T08:33:15.743607+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:15.743607+0000","last_undegraded":"2024-04-22T08:33:15.743607+0000","last_fullsized":"2024-04-22T08:33:15.743607+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28651520,"num_objects":6995,"num_object_clones":0,"num_object_copies":13990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13026","reported_seq":13471,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.719635+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:15.719635+0000","last_peered":"2024-04-22T08:33:15.719635+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:15.719635+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28004376,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9853,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":663},{"shards":"2","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13253","reported_seq":13286,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.700350+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:15.700350+0000","last_peered":"2024-04-22T08:33:15.700350+0000","last_clean":"2024-04-22T08:33:15.700350+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:15.700350+0000","last_undegraded":"2024-04-22T08:33:15.700350+0000","last_fullsized":"2024-04-22T08:33:15.700350+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28102656,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13253,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13080","reported_seq":13221,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.830481+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:15.830481+0000","last_peered":"2024-04-22T08:33:15.830481+0000","last_clean":"2024-04-22T08:33:15.830481+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:15.830481+0000","last_undegraded":"2024-04-22T08:33:15.830481+0000","last_fullsized":"2024-04-22T08:33:15.830481+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10172,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13230","reported_seq":16244,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.883714+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:15.883714+0000","last_peered":"2024-04-22T08:33:15.883714+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:15.883714+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27922432,"num_objects":6817,"num_object_clones":0,"num_object_copies":13634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1782,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":6817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":4972544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13170","reported_seq":20909,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:19.683923+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:19.683923+0000","last_peered":"2024-04-22T08:33:19.683923+0000","last_clean":"2024-04-22T08:33:19.683923+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:19.683923+0000","last_undegraded":"2024-04-22T08:33:19.683923+0000","last_fullsized":"2024-04-22T08:33:19.683923+0000","mapping_epoch":36,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27418624,"num_objects":6694,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13013","reported_seq":13160,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.777808+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:15.777808+0000","last_peered":"2024-04-22T08:33:15.777808+0000","last_clean":"2024-04-22T08:33:15.777808+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:15.777808+0000","last_undegraded":"2024-04-22T08:33:15.777808+0000","last_fullsized":"2024-04-22T08:33:15.777808+0000","mapping_epoch":27,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13348","reported_seq":13752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.750610+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:15.750610+0000","last_peered":"2024-04-22T08:33:15.750610+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:15.750610+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28153042,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":683},{"shards":"2","objects":3126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.121959+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:18.121959+0000","last_peered":"2024-04-22T08:32:18.121959+0000","last_clean":"2024-04-22T08:32:18.121959+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:18.121959+0000","last_undegraded":"2024-04-22T08:32:18.121959+0000","last_fullsized":"2024-04-22T08:32:18.121959+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13217","reported_seq":13954,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.848338+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:15.848338+0000","last_peered":"2024-04-22T08:33:15.848338+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:15.848338+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546258,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13215,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3528},{"shards":"0,2","objects":651}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13046","reported_seq":13529,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.602171+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:15.602171+0000","last_peered":"2024-04-22T08:33:15.602171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:15.602171+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28192768,"num_objects":6883,"num_object_clones":0,"num_object_copies":13766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":671},{"shards":"2","objects":3300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13115","reported_seq":20793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:15.625198+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:15.625198+0000","last_peered":"2024-04-22T08:33:15.625198+0000","last_clean":"2024-04-22T08:33:15.625198+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:15.625198+0000","last_undegraded":"2024-04-22T08:33:15.625198+0000","last_fullsized":"2024-04-22T08:33:15.625198+0000","mapping_epoch":38,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27931858,"num_objects":6821,"num_object_clones":0,"num_object_copies":13642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13122","reported_seq":13756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.831939+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:15.831939+0000","last_peered":"2024-04-22T08:33:15.831939+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:15.831939+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28754537,"num_objects":7021,"num_object_clones":0,"num_object_copies":14042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13121,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3576},{"shards":"0,2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13031","reported_seq":13617,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.881591+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:15.881591+0000","last_peered":"2024-04-22T08:33:15.881591+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:15.881591+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3000","ondisk_log_start":"28'3000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27521024,"num_objects":6719,"num_object_clones":0,"num_object_copies":13438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3086},{"shards":"0,2","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.126690+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:18.126690+0000","last_peered":"2024-04-22T08:32:18.126690+0000","last_clean":"2024-04-22T08:32:18.126690+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:18.126690+0000","last_undegraded":"2024-04-22T08:32:18.126690+0000","last_fullsized":"2024-04-22T08:32:18.126690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13109","reported_seq":13616,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.877373+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:15.877373+0000","last_peered":"2024-04-22T08:33:15.877373+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:15.877373+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13107,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3440},{"shards":"0,2","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13227","reported_seq":13860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:15.842629+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:15.842629+0000","last_peered":"2024-04-22T08:33:15.842629+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:15.842629+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27752868,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3141},{"shards":"0,2","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":449675117,"num_objects":109796,"num_object_clones":0,"num_object_copies":219592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33897,"num_objects_misplaced":675,"num_objects_unfound":0,"num_objects_dirty":109796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183908,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11035,"num_bytes_recovered":36049513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1058848768,"data_stored":1058774523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160533,"ondisk_log_size":160533,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117071,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652728,"kb_used_data":342192,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93719112,"statfs":{"total":96636764160,"available":95968370688,"internally_reserved":0,"allocated":350404608,"data_stored":350178681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":317976271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,6,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":7.556}]}]},{"osd":1,"up_from":33,"seq":141733920834,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637124,"kb_used_data":302776,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93734716,"statfs":{"total":96636764160,"available":95984349184,"internally_reserved":0,"allocated":310042624,"data_stored":309761033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":342355660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.379}]}]},{"osd":0,"up_from":21,"seq":90194313299,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774468,"kb_used_data":391804,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93597372,"statfs":{"total":96636764160,"available":95843708928,"internally_reserved":0,"allocated":401207296,"data_stored":400991903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,2,4,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399941632,"data_stored":399938153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309633024,"data_stored":309569809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349274112,"data_stored":349266561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'13209", 'reported_seq': 13817, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.712687+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:15.712687+0000', 'last_peered': '2024-04-22T08:33:15.712687+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:15.712687+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28504064, 'num_objects': 6959, 'num_object_clones': 0, 'num_object_copies': 13918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3523, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6959, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13209, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 271, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3436}, {'shards': '0,2', 'objects': 669}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13026", 'reported_seq': 13471, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.719635+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:15.719635+0000', 'last_peered': '2024-04-22T08:33:15.719635+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:15.719635+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28004376, 'num_objects': 6838, 'num_object_clones': 0, 'num_object_copies': 13676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6838, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9853, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 161, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 663}, {'shards': '2', 'objects': 3244}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:22.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'13230", 'reported_seq': 16244, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.883714+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:15.883714+0000', 'last_peered': '2024-04-22T08:33:15.883714+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:15.883714+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27922432, 'num_objects': 6817, 'num_object_clones': 0, 'num_object_copies': 13634, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1782, 'num_objects_misplaced': 675, 'num_objects_unfound': 0, 'num_objects_dirty': 6817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10099, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1425, 'num_bytes_recovered': 4972544, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4360}, {'shards': '0,2', 'objects': 675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13348", 'reported_seq': 13752, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.750610+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:15.750610+0000', 'last_peered': '2024-04-22T08:33:15.750610+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:15.750610+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "28'3300", 'ondisk_log_start': "28'3300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28153042, 'num_objects': 6875, 'num_object_clones': 0, 'num_object_copies': 13750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3751, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13288, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 144, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 683}, {'shards': '2', 'objects': 3126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'13217", 'reported_seq': 13954, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.848338+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:15.848338+0000', 'last_peered': '2024-04-22T08:33:15.848338+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:15.848338+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28546258, 'num_objects': 6971, 'num_object_clones': 0, 'num_object_copies': 13942, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3445, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13215, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 331, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3528}, {'shards': '0,2', 'objects': 651}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13046", 'reported_seq': 13529, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.602171+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:15.602171+0000', 'last_peered': '2024-04-22T08:33:15.602171+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:15.602171+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28192768, 'num_objects': 6883, 'num_object_clones': 0, 'num_object_copies': 13766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3585, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9500, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 167, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 671}, {'shards': '2', 'objects': 3300}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'13122", 'reported_seq': 13756, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.831939+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:15.831939+0000', 'last_peered': '2024-04-22T08:33:15.831939+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:15.831939+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "28'3100", 'ondisk_log_start': "28'3100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28754537, 'num_objects': 7021, 'num_object_clones': 0, 'num_object_copies': 14042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13121, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 278, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3576}, {'shards': '0,2', 'objects': 671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:22.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13031", 'reported_seq': 13617, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.881591+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:15.881591+0000', 'last_peered': '2024-04-22T08:33:15.881591+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:15.881591+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'3000", 'ondisk_log_start': "28'3000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27521024, 'num_objects': 6719, 'num_object_clones': 0, 'num_object_copies': 13438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3636, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9262, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 204, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3086}, {'shards': '0,2', 'objects': 685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:22.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'13109", 'reported_seq': 13616, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.877373+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:15.877373+0000', 'last_peered': '2024-04-22T08:33:15.877373+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:15.877373+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'3100", 'ondisk_log_start': "28'3100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28382418, 'num_objects': 6931, 'num_object_clones': 0, 'num_object_copies': 13862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3493, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13107, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 216, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3440}, {'shards': '0,2', 'objects': 617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:22.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'13227", 'reported_seq': 13860, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:15.842629+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:15.842629+0000', 'last_peered': '2024-04-22T08:33:15.842629+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:15.842629+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27752868, 'num_objects': 6779, 'num_object_clones': 0, 'num_object_copies': 13558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3641, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13224, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 248, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3141}, {'shards': '0,2', 'objects': 661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:22.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:22.751 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33847/215622 objects degraded (15.697%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":437,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":107811,"data_bytes":441544557,"bytes_used":2077241344,"bytes_avail":287833051136,"bytes_total":289910292480,"degraded_objects":33847,"degraded_total":215622,"degraded_ratio":0.15697377818589939,"misplaced_objects":512,"misplaced_total":215622,"misplaced_ratio":0.002374525790503752,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":121810,"recovering_keys_per_sec":0,"num_objects_recovered":607,"num_bytes_recovered":1462272,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":513},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-22T08:32:22.266936+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:22.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:22.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:23.291 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:23.292 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:23.330 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-04-22T08:33:22.300407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441544557,"num_objects":107811,"num_object_clones":0,"num_object_copies":215622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33847,"num_objects_misplaced":512,"num_objects_unfound":0,"num_objects_dirty":107811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185893,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11248,"num_bytes_recovered":36565609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2028556,"kb_used_data":1025580,"kb_used_omap":12,"kb_used_meta":1002931,"kb_avail":281086964,"statfs":{"total":289910292480,"available":287833051136,"internally_reserved":0,"allocated":1050193920,"data_stored":1049471009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1027001454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,5,6,9,0,0,3,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25264128,"num_objects":-6168,"num_object_clones":0,"num_object_copies":-12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":-430,"num_objects_unfound":0,"num_objects_dirty":-6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004488"},"pg_stats":[{"pgid":"2.f","version":"39'13346","reported_seq":13978,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.909182+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:20.909182+0000","last_peered":"2024-04-22T08:33:20.909182+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:20.909182+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27942912,"num_objects":6822,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13373","reported_seq":13491,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:20.759119+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:20.759119+0000","last_peered":"2024-04-22T08:33:20.759119+0000","last_clean":"2024-04-22T08:33:20.759119+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:20.759119+0000","last_undegraded":"2024-04-22T08:33:20.759119+0000","last_fullsized":"2024-04-22T08:33:20.759119+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28094464,"num_objects":6859,"num_object_clones":0,"num_object_copies":13718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13146","reported_seq":13611,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.716343+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:20.716343+0000","last_peered":"2024-04-22T08:33:20.716343+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:20.716343+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27512856,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9973,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":543},{"shards":"2","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13391","reported_seq":13424,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:20.774283+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:20.774283+0000","last_peered":"2024-04-22T08:33:20.774283+0000","last_clean":"2024-04-22T08:33:20.774283+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:20.774283+0000","last_undegraded":"2024-04-22T08:33:20.774283+0000","last_fullsized":"2024-04-22T08:33:20.774283+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27537408,"num_objects":6723,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13391,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13240","reported_seq":13381,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:20.861317+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:20.861317+0000","last_peered":"2024-04-22T08:33:20.861317+0000","last_clean":"2024-04-22T08:33:20.861317+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:20.861317+0000","last_undegraded":"2024-04-22T08:33:20.861317+0000","last_fullsized":"2024-04-22T08:33:20.861317+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27357184,"num_objects":6679,"num_object_clones":0,"num_object_copies":13358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10332,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13372","reported_seq":16650,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.923580+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:20.923580+0000","last_peered":"2024-04-22T08:33:20.923580+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:20.923580+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27340800,"num_objects":6675,"num_object_clones":0,"num_object_copies":13350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":512,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1557,"num_bytes_recovered":5488640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4350},{"shards":"0,2","objects":512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13170","reported_seq":20909,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:19.683923+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:19.683923+0000","last_peered":"2024-04-22T08:33:19.683923+0000","last_clean":"2024-04-22T08:33:19.683923+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:19.683923+0000","last_undegraded":"2024-04-22T08:33:19.683923+0000","last_fullsized":"2024-04-22T08:33:19.683923+0000","mapping_epoch":36,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27418624,"num_objects":6694,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13162","reported_seq":13310,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:20.908270+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:20.908270+0000","last_peered":"2024-04-22T08:33:20.908270+0000","last_clean":"2024-04-22T08:33:20.908270+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:20.908270+0000","last_undegraded":"2024-04-22T08:33:20.908270+0000","last_fullsized":"2024-04-22T08:33:20.908270+0000","mapping_epoch":27,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27213824,"num_objects":6644,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13489","reported_seq":13913,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.731436+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:20.731436+0000","last_peered":"2024-04-22T08:33:20.731436+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:20.731436+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"28'3400","ondisk_log_start":"28'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27575506,"num_objects":6734,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13429,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":542},{"shards":"2","objects":2995}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13343","reported_seq":14116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.886999+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:20.886999+0000","last_peered":"2024-04-22T08:33:20.886999+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:20.886999+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030162,"num_objects":6845,"num_object_clones":0,"num_object_copies":13690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13341,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3420},{"shards":"0,2","objects":525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13170","reported_seq":13658,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.747156+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:20.747156+0000","last_peered":"2024-04-22T08:33:20.747156+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:20.747156+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27684864,"num_objects":6759,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":547},{"shards":"2","objects":3179}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13234","reported_seq":20912,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:20.675135+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:20.675135+0000","last_peered":"2024-04-22T08:33:20.675135+0000","last_clean":"2024-04-22T08:33:20.675135+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:20.675135+0000","last_undegraded":"2024-04-22T08:33:20.675135+0000","last_fullsized":"2024-04-22T08:33:20.675135+0000","mapping_epoch":38,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27444434,"num_objects":6702,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13232,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13233","reported_seq":13885,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.910197+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:20.910197+0000","last_peered":"2024-04-22T08:33:20.910197+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:20.910197+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28299881,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13232,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3474},{"shards":"0,2","objects":560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13159","reported_seq":13760,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.794695+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:20.794695+0000","last_peered":"2024-04-22T08:33:20.794695+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:20.794695+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3100","ondisk_log_start":"28'3100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26996736,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9390,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2966},{"shards":"0,2","objects":557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13236","reported_seq":13750,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.929269+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:20.929269+0000","last_peered":"2024-04-22T08:33:20.929269+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:20.929269+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27862226,"num_objects":6804,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13234,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3316},{"shards":"0,2","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13354","reported_seq":14007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:20.862312+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:20.862312+0000","last_peered":"2024-04-22T08:33:20.862312+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:20.862312+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27232676,"num_objects":6652,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13351,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3024},{"shards":"0,2","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":441544557,"num_objects":107811,"num_object_clones":0,"num_object_copies":215622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33847,"num_objects_misplaced":512,"num_objects_unfound":0,"num_objects_dirty":107811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185893,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11248,"num_bytes_recovered":36565609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1047793664,"data_stored":1047719419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117072,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635388,"kb_used_data":337136,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93736452,"statfs":{"total":96636764160,"available":95986126848,"internally_reserved":0,"allocated":345227264,"data_stored":345001337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":3.617},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.24399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920834,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637124,"kb_used_data":302776,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93734716,"statfs":{"total":96636764160,"available":95984349184,"internally_reserved":0,"allocated":310042624,"data_stored":309761033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":342355660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.379}]}]},{"osd":0,"up_from":21,"seq":90194313300,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756044,"kb_used_data":385668,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93615796,"statfs":{"total":96636764160,"available":95862575104,"internally_reserved":0,"allocated":394924032,"data_stored":394708639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,2,4,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393314304,"data_stored":393310825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309633024,"data_stored":309569809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344846336,"data_stored":344838785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:26.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:26.963 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:26.964 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:27.019 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-04-22T08:33:26.301268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033808,"kb_used_data":1014448,"kb_used_omap":12,"kb_used_meta":1019315,"kb_avail":281081712,"statfs":{"total":289910292480,"available":287827673088,"internally_reserved":0,"allocated":1038794752,"data_stored":1038071841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1043778670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,1,4,3,1,0,13],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25505792,"num_objects":-6227,"num_object_clones":0,"num_object_copies":-12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":-6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":1499136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"2.f","version":"39'13480","reported_seq":14128,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.939502+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:25.939502+0000","last_peered":"2024-04-22T08:33:25.939502+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:25.939502+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3186},{"shards":"0,2","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13512","reported_seq":13630,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.781486+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:25.781486+0000","last_peered":"2024-04-22T08:33:25.781486+0000","last_clean":"2024-04-22T08:33:25.781486+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:25.781486+0000","last_undegraded":"2024-04-22T08:33:25.781486+0000","last_fullsized":"2024-04-22T08:33:25.781486+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27525120,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13275","reported_seq":13760,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.770273+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:25.770273+0000","last_peered":"2024-04-22T08:33:25.770273+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:25.770273+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26984472,"num_objects":6589,"num_object_clones":0,"num_object_copies":13178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10102,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":414},{"shards":"2","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13518","reported_seq":13551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.760917+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:25.760917+0000","last_peered":"2024-04-22T08:33:25.760917+0000","last_clean":"2024-04-22T08:33:25.760917+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:25.760917+0000","last_undegraded":"2024-04-22T08:33:25.760917+0000","last_fullsized":"2024-04-22T08:33:25.760917+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":6596,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13372","reported_seq":13513,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.892762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:25.892762+0000","last_peered":"2024-04-22T08:33:25.892762+0000","last_clean":"2024-04-22T08:33:25.892762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:25.892762+0000","last_undegraded":"2024-04-22T08:33:25.892762+0000","last_fullsized":"2024-04-22T08:33:25.892762+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26816512,"num_objects":6547,"num_object_clones":0,"num_object_copies":13094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13512","reported_seq":17051,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.875963+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:25.875963+0000","last_peered":"2024-04-22T08:33:25.875963+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:25.875963+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767360,"num_objects":6535,"num_object_clones":0,"num_object_copies":13070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":6535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":6025216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4341},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13308","reported_seq":21047,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:24.703417+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:24.703417+0000","last_peered":"2024-04-22T08:33:24.703417+0000","last_clean":"2024-04-22T08:33:24.703417+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:24.703417+0000","last_undegraded":"2024-04-22T08:33:24.703417+0000","last_fullsized":"2024-04-22T08:33:24.703417+0000","mapping_epoch":36,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26853376,"num_objects":6556,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13273","reported_seq":13421,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.855558+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:25.855558+0000","last_peered":"2024-04-22T08:33:25.855558+0000","last_clean":"2024-04-22T08:33:25.855558+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:25.855558+0000","last_undegraded":"2024-04-22T08:33:25.855558+0000","last_fullsized":"2024-04-22T08:33:25.855558+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13607","reported_seq":14042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.773042+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:25.773042+0000","last_peered":"2024-04-22T08:33:25.773042+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:25.773042+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27092178,"num_objects":6616,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13547,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":424},{"shards":"2","objects":2883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13456","reported_seq":14237,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.873280+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:25.873280+0000","last_peered":"2024-04-22T08:33:25.873280+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:25.873280+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27567314,"num_objects":6732,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13454,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13299","reported_seq":13811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.665134+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:25.665134+0000","last_peered":"2024-04-22T08:33:25.665134+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:25.665134+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":6630,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":418},{"shards":"2","objects":3061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13361","reported_seq":21039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.747542+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:25.747542+0000","last_peered":"2024-04-22T08:33:25.747542+0000","last_clean":"2024-04-22T08:33:25.747542+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:25.747542+0000","last_undegraded":"2024-04-22T08:33:25.747542+0000","last_fullsized":"2024-04-22T08:33:25.747542+0000","mapping_epoch":38,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26924242,"num_objects":6575,"num_object_clones":0,"num_object_copies":13150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13357","reported_seq":14019,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.906824+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:25.906824+0000","last_peered":"2024-04-22T08:33:25.906824+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:25.906824+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3300","ondisk_log_start":"31'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27791977,"num_objects":6786,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13356,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13281","reported_seq":13889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.913003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:25.913003+0000","last_peered":"2024-04-22T08:33:25.913003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:25.913003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26497024,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13350","reported_seq":13886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.945872+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:25.945872+0000","last_peered":"2024-04-22T08:33:25.945872+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:25.945872+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27395282,"num_objects":6690,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13348,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13498","reported_seq":14173,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.895196+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:25.895196+0000","last_peered":"2024-04-22T08:33:25.895196+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:25.895196+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3400","ondisk_log_start":"28'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26642852,"num_objects":6508,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13495,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2891},{"shards":"0,2","objects":390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036320768,"data_stored":1036246523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635472,"kb_used_data":333120,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93736368,"statfs":{"total":96636764160,"available":95986040832,"internally_reserved":0,"allocated":341114880,"data_stored":340888953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,1,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27200000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920835,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644272,"kb_used_data":301736,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93727568,"statfs":{"total":96636764160,"available":95977029632,"internally_reserved":0,"allocated":308977664,"data_stored":308696073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.504}]}]},{"osd":0,"up_from":21,"seq":90194313301,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754064,"kb_used_data":379592,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93617776,"statfs":{"total":96636764160,"available":95864602624,"internally_reserved":0,"allocated":388702208,"data_stored":388486815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,1,2,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387084288,"data_stored":387080809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308547584,"data_stored":308484369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340688896,"data_stored":340681345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:27.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:27.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:27.726 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:27.772 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-04-22T08:33:26.301268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033808,"kb_used_data":1014448,"kb_used_omap":12,"kb_used_meta":1019315,"kb_avail":281081712,"statfs":{"total":289910292480,"available":287827673088,"internally_reserved":0,"allocated":1038794752,"data_stored":1038071841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1043778670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,1,4,3,1,0,13],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25505792,"num_objects":-6227,"num_object_clones":0,"num_object_copies":-12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":-6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":1499136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"2.f","version":"39'13480","reported_seq":14128,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.939502+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:25.939502+0000","last_peered":"2024-04-22T08:33:25.939502+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:25.939502+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3186},{"shards":"0,2","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13512","reported_seq":13630,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.781486+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:25.781486+0000","last_peered":"2024-04-22T08:33:25.781486+0000","last_clean":"2024-04-22T08:33:25.781486+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:25.781486+0000","last_undegraded":"2024-04-22T08:33:25.781486+0000","last_fullsized":"2024-04-22T08:33:25.781486+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27525120,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13275","reported_seq":13760,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.770273+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:25.770273+0000","last_peered":"2024-04-22T08:33:25.770273+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:25.770273+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26984472,"num_objects":6589,"num_object_clones":0,"num_object_copies":13178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10102,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":414},{"shards":"2","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13518","reported_seq":13551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.760917+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:25.760917+0000","last_peered":"2024-04-22T08:33:25.760917+0000","last_clean":"2024-04-22T08:33:25.760917+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:25.760917+0000","last_undegraded":"2024-04-22T08:33:25.760917+0000","last_fullsized":"2024-04-22T08:33:25.760917+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":6596,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13372","reported_seq":13513,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.892762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:25.892762+0000","last_peered":"2024-04-22T08:33:25.892762+0000","last_clean":"2024-04-22T08:33:25.892762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:25.892762+0000","last_undegraded":"2024-04-22T08:33:25.892762+0000","last_fullsized":"2024-04-22T08:33:25.892762+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26816512,"num_objects":6547,"num_object_clones":0,"num_object_copies":13094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13512","reported_seq":17051,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.875963+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:25.875963+0000","last_peered":"2024-04-22T08:33:25.875963+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:25.875963+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767360,"num_objects":6535,"num_object_clones":0,"num_object_copies":13070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":6535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":6025216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4341},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13308","reported_seq":21047,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:24.703417+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:24.703417+0000","last_peered":"2024-04-22T08:33:24.703417+0000","last_clean":"2024-04-22T08:33:24.703417+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:24.703417+0000","last_undegraded":"2024-04-22T08:33:24.703417+0000","last_fullsized":"2024-04-22T08:33:24.703417+0000","mapping_epoch":36,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26853376,"num_objects":6556,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13273","reported_seq":13421,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.855558+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:25.855558+0000","last_peered":"2024-04-22T08:33:25.855558+0000","last_clean":"2024-04-22T08:33:25.855558+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:25.855558+0000","last_undegraded":"2024-04-22T08:33:25.855558+0000","last_fullsized":"2024-04-22T08:33:25.855558+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13607","reported_seq":14042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.773042+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:25.773042+0000","last_peered":"2024-04-22T08:33:25.773042+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:25.773042+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27092178,"num_objects":6616,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13547,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":424},{"shards":"2","objects":2883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13456","reported_seq":14237,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.873280+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:25.873280+0000","last_peered":"2024-04-22T08:33:25.873280+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:25.873280+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27567314,"num_objects":6732,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13454,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13299","reported_seq":13811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.665134+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:25.665134+0000","last_peered":"2024-04-22T08:33:25.665134+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:25.665134+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":6630,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":418},{"shards":"2","objects":3061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13361","reported_seq":21039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.747542+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:25.747542+0000","last_peered":"2024-04-22T08:33:25.747542+0000","last_clean":"2024-04-22T08:33:25.747542+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:25.747542+0000","last_undegraded":"2024-04-22T08:33:25.747542+0000","last_fullsized":"2024-04-22T08:33:25.747542+0000","mapping_epoch":38,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26924242,"num_objects":6575,"num_object_clones":0,"num_object_copies":13150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13357","reported_seq":14019,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.906824+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:25.906824+0000","last_peered":"2024-04-22T08:33:25.906824+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:25.906824+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3300","ondisk_log_start":"31'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27791977,"num_objects":6786,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13356,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13281","reported_seq":13889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.913003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:25.913003+0000","last_peered":"2024-04-22T08:33:25.913003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:25.913003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26497024,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13350","reported_seq":13886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.945872+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:25.945872+0000","last_peered":"2024-04-22T08:33:25.945872+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:25.945872+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27395282,"num_objects":6690,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13348,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13498","reported_seq":14173,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.895196+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:25.895196+0000","last_peered":"2024-04-22T08:33:25.895196+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:25.895196+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3400","ondisk_log_start":"28'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26642852,"num_objects":6508,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13495,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2891},{"shards":"0,2","objects":390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036320768,"data_stored":1036246523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635472,"kb_used_data":333120,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93736368,"statfs":{"total":96636764160,"available":95986040832,"internally_reserved":0,"allocated":341114880,"data_stored":340888953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,1,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27200000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920835,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644272,"kb_used_data":301736,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93727568,"statfs":{"total":96636764160,"available":95977029632,"internally_reserved":0,"allocated":308977664,"data_stored":308696073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.504}]}]},{"osd":0,"up_from":21,"seq":90194313301,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754064,"kb_used_data":379592,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93617776,"statfs":{"total":96636764160,"available":95864602624,"internally_reserved":0,"allocated":388702208,"data_stored":388486815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,1,2,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387084288,"data_stored":387080809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308547584,"data_stored":308484369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340688896,"data_stored":340681345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:27.774 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:27.774 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'13480", 'reported_seq': 14128, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.939502+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:25.939502+0000', 'last_peered': '2024-04-22T08:33:25.939502+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:25.939502+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27394048, 'num_objects': 6688, 'num_object_clones': 0, 'num_object_copies': 13376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3502, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13480, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 292, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3186}, {'shards': '0,2', 'objects': 398}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.774 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:27.774 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13275", 'reported_seq': 13760, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.770273+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:25.770273+0000', 'last_peered': '2024-04-22T08:33:25.770273+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:25.770273+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26984472, 'num_objects': 6589, 'num_object_clones': 0, 'num_object_copies': 13178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3577, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6589, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10102, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 179, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 414}, {'shards': '2', 'objects': 3013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'13512", 'reported_seq': 17051, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.875963+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:25.875963+0000', 'last_peered': '2024-04-22T08:33:25.875963+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:25.875963+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "31'3500", 'ondisk_log_start': "31'3500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26767360, 'num_objects': 6535, 'num_object_clones': 0, 'num_object_copies': 13070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 380, 'num_objects_unfound': 0, 'num_objects_dirty': 6535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10381, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1688, 'num_bytes_recovered': 6025216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4341}, {'shards': '0,2', 'objects': 380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13607", 'reported_seq': 14042, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.773042+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:25.773042+0000', 'last_peered': '2024-04-22T08:33:25.773042+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:25.773042+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "31'3600", 'ondisk_log_start': "31'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27092178, 'num_objects': 6616, 'num_object_clones': 0, 'num_object_copies': 13232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3735, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13547, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 160, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 424}, {'shards': '2', 'objects': 2883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'13456", 'reported_seq': 14237, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.873280+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:25.873280+0000', 'last_peered': '2024-04-22T08:33:25.873280+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:25.873280+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27567314, 'num_objects': 6732, 'num_object_clones': 0, 'num_object_copies': 13464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3423, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13454, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 353, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3311}, {'shards': '0,2', 'objects': 412}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13299", 'reported_seq': 13811, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.665134+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:25.665134+0000', 'last_peered': '2024-04-22T08:33:25.665134+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:25.665134+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27156480, 'num_objects': 6630, 'num_object_clones': 0, 'num_object_copies': 13260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3571, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9753, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 181, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 418}, {'shards': '2', 'objects': 3061}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:27.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'13357", 'reported_seq': 14019, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.906824+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:25.906824+0000', 'last_peered': '2024-04-22T08:33:25.906824+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:25.906824+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "31'3300", 'ondisk_log_start': "31'3300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27791977, 'num_objects': 6786, 'num_object_clones': 0, 'num_object_copies': 13572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3432, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13356, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 292, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3355}, {'shards': '0,2', 'objects': 436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13281", 'reported_seq': 13889, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.913003+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:25.913003+0000', 'last_peered': '2024-04-22T08:33:25.913003+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:25.913003+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "28'3200", 'ondisk_log_start': "28'3200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26497024, 'num_objects': 6469, 'num_object_clones': 0, 'num_object_copies': 12938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3625, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9512, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 215, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2847}, {'shards': '0,2', 'objects': 435}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'13350", 'reported_seq': 13886, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.945872+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:25.945872+0000', 'last_peered': '2024-04-22T08:33:25.945872+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:25.945872+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "28'3300", 'ondisk_log_start': "28'3300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27395282, 'num_objects': 6690, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3479, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13348, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 230, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3213}, {'shards': '0,2', 'objects': 376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:27.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'13498", 'reported_seq': 14173, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:25.895196+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:25.895196+0000', 'last_peered': '2024-04-22T08:33:25.895196+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:25.895196+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "28'3400", 'ondisk_log_start': "28'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26642852, 'num_objects': 6508, 'num_object_clones': 0, 'num_object_copies': 13016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3620, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13495, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 269, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2891}, {'shards': '0,2', 'objects': 390}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:27.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:28.453 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33778/211540 objects degraded (15.968%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":443,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":105770,"data_bytes":433184621,"bytes_used":2082619392,"bytes_avail":287827673088,"bytes_total":289910292480,"degraded_objects":33778,"degraded_total":211540,"degraded_ratio":0.15967665689704075,"misplaced_objects":380,"misplaced_total":211540,"misplaced_ratio":0.00179635057199584,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":124882,"recovering_keys_per_sec":0,"num_objects_recovered":595,"num_bytes_recovered":1499136,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":518},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:28.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:28.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:29.248 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:29.249 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:29.301 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-04-22T08:33:28.304329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033808,"kb_used_data":1014448,"kb_used_omap":12,"kb_used_meta":1019315,"kb_avail":281081712,"statfs":{"total":289910292480,"available":287827673088,"internally_reserved":0,"allocated":1038794752,"data_stored":1038071841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1043778670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,1,4,3,1,0,13],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17002496,"num_objects":-4151,"num_object_clones":0,"num_object_copies":-8302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":-295,"num_objects_unfound":0,"num_objects_dirty":-4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":1052672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006965"},"pg_stats":[{"pgid":"2.f","version":"39'13480","reported_seq":14128,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.939502+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:25.939502+0000","last_peered":"2024-04-22T08:33:25.939502+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:25.939502+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3186},{"shards":"0,2","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13512","reported_seq":13630,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.781486+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:25.781486+0000","last_peered":"2024-04-22T08:33:25.781486+0000","last_clean":"2024-04-22T08:33:25.781486+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:25.781486+0000","last_undegraded":"2024-04-22T08:33:25.781486+0000","last_fullsized":"2024-04-22T08:33:25.781486+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27525120,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13275","reported_seq":13760,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.770273+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:25.770273+0000","last_peered":"2024-04-22T08:33:25.770273+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:25.770273+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26984472,"num_objects":6589,"num_object_clones":0,"num_object_copies":13178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10102,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":414},{"shards":"2","objects":3013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13518","reported_seq":13551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.760917+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:25.760917+0000","last_peered":"2024-04-22T08:33:25.760917+0000","last_clean":"2024-04-22T08:33:25.760917+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:25.760917+0000","last_undegraded":"2024-04-22T08:33:25.760917+0000","last_fullsized":"2024-04-22T08:33:25.760917+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":6596,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13372","reported_seq":13513,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.892762+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:25.892762+0000","last_peered":"2024-04-22T08:33:25.892762+0000","last_clean":"2024-04-22T08:33:25.892762+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:25.892762+0000","last_undegraded":"2024-04-22T08:33:25.892762+0000","last_fullsized":"2024-04-22T08:33:25.892762+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26816512,"num_objects":6547,"num_object_clones":0,"num_object_copies":13094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13512","reported_seq":17051,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.875963+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:25.875963+0000","last_peered":"2024-04-22T08:33:25.875963+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:25.875963+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26767360,"num_objects":6535,"num_object_clones":0,"num_object_copies":13070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":6535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":6025216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4341},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13308","reported_seq":21047,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:24.703417+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:24.703417+0000","last_peered":"2024-04-22T08:33:24.703417+0000","last_clean":"2024-04-22T08:33:24.703417+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:24.703417+0000","last_undegraded":"2024-04-22T08:33:24.703417+0000","last_fullsized":"2024-04-22T08:33:24.703417+0000","mapping_epoch":36,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26853376,"num_objects":6556,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13273","reported_seq":13421,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.855558+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:25.855558+0000","last_peered":"2024-04-22T08:33:25.855558+0000","last_clean":"2024-04-22T08:33:25.855558+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:25.855558+0000","last_undegraded":"2024-04-22T08:33:25.855558+0000","last_fullsized":"2024-04-22T08:33:25.855558+0000","mapping_epoch":27,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13607","reported_seq":14042,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.773042+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:25.773042+0000","last_peered":"2024-04-22T08:33:25.773042+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:25.773042+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27092178,"num_objects":6616,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13547,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":424},{"shards":"2","objects":2883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13456","reported_seq":14237,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.873280+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:25.873280+0000","last_peered":"2024-04-22T08:33:25.873280+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:25.873280+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27567314,"num_objects":6732,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13454,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13299","reported_seq":13811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.665134+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:25.665134+0000","last_peered":"2024-04-22T08:33:25.665134+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:25.665134+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":6630,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":418},{"shards":"2","objects":3061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13361","reported_seq":21039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:25.747542+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:25.747542+0000","last_peered":"2024-04-22T08:33:25.747542+0000","last_clean":"2024-04-22T08:33:25.747542+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:25.747542+0000","last_undegraded":"2024-04-22T08:33:25.747542+0000","last_fullsized":"2024-04-22T08:33:25.747542+0000","mapping_epoch":38,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26924242,"num_objects":6575,"num_object_clones":0,"num_object_copies":13150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13357","reported_seq":14019,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.906824+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:25.906824+0000","last_peered":"2024-04-22T08:33:25.906824+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:25.906824+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3300","ondisk_log_start":"31'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27791977,"num_objects":6786,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13356,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13281","reported_seq":13889,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.913003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:25.913003+0000","last_peered":"2024-04-22T08:33:25.913003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:25.913003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"28'3200","ondisk_log_start":"28'3200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26497024,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13350","reported_seq":13886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.945872+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:25.945872+0000","last_peered":"2024-04-22T08:33:25.945872+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:25.945872+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27395282,"num_objects":6690,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13348,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13498","reported_seq":14173,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:25.895196+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:25.895196+0000","last_peered":"2024-04-22T08:33:25.895196+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:25.895196+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"28'3400","ondisk_log_start":"28'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26642852,"num_objects":6508,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13495,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2891},{"shards":"0,2","objects":390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":433184621,"num_objects":105770,"num_object_clones":0,"num_object_copies":211540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33778,"num_objects_misplaced":380,"num_objects_unfound":0,"num_objects_dirty":105770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187934,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11449,"num_bytes_recovered":37102185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036320768,"data_stored":1036246523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635472,"kb_used_data":333120,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93736368,"statfs":{"total":96636764160,"available":95986040832,"internally_reserved":0,"allocated":341114880,"data_stored":340888953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,1,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27200000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920835,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644272,"kb_used_data":301736,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93727568,"statfs":{"total":96636764160,"available":95977029632,"internally_reserved":0,"allocated":308977664,"data_stored":308696073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.504}]}]},{"osd":0,"up_from":21,"seq":90194313301,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754064,"kb_used_data":379592,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93617776,"statfs":{"total":96636764160,"available":95864602624,"internally_reserved":0,"allocated":388702208,"data_stored":388486815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,1,2,0,0,11],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387084288,"data_stored":387080809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308547584,"data_stored":308484369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340688896,"data_stored":340681345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:32.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:32.941 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:32.979 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-04-22T08:33:32.305268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2038532,"kb_used_data":1002796,"kb_used_omap":12,"kb_used_meta":1035699,"kb_avail":281076988,"statfs":{"total":289910292480,"available":287822835712,"internally_reserved":0,"allocated":1026863104,"data_stored":1026140193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1060555886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,2,3,2,3,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25079808,"num_objects":-6123,"num_object_clones":0,"num_object_copies":-12246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":1544192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005333"},"pg_stats":[{"pgid":"2.f","version":"39'13607","reported_seq":14283,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.969493+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:30.969493+0000","last_peered":"2024-04-22T08:33:30.969493+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:30.969493+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26873856,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13607,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3072},{"shards":"0,2","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13631","reported_seq":13749,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.711100+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:30.711100+0000","last_peered":"2024-04-22T08:33:30.711100+0000","last_clean":"2024-04-22T08:33:30.711100+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:30.711100+0000","last_undegraded":"2024-04-22T08:33:30.711100+0000","last_fullsized":"2024-04-22T08:33:30.711100+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27037696,"num_objects":6601,"num_object_clones":0,"num_object_copies":13202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13410","reported_seq":13909,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.464468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:30.464468+0000","last_peered":"2024-04-22T08:33:30.464468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:30.464468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431512,"num_objects":6454,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":279},{"shards":"2","objects":2885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13665","reported_seq":13698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.798237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:30.798237+0000","last_peered":"2024-04-22T08:33:30.798237+0000","last_clean":"2024-04-22T08:33:30.798237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:30.798237+0000","last_undegraded":"2024-04-22T08:33:30.798237+0000","last_fullsized":"2024-04-22T08:33:30.798237+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26415104,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13665,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13499","reported_seq":13641,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.935797+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:30.935797+0000","last_peered":"2024-04-22T08:33:30.935797+0000","last_clean":"2024-04-22T08:33:30.935797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:30.935797+0000","last_undegraded":"2024-04-22T08:33:30.935797+0000","last_fullsized":"2024-04-22T08:33:30.935797+0000","mapping_epoch":27,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26296320,"num_objects":6420,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13639","reported_seq":17419,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.988075+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:30.988075+0000","last_peered":"2024-04-22T08:33:30.988075+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:30.988075+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":6408,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":6408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":6516736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4334},{"shards":"0,2","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13418","reported_seq":21157,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:29.710498+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:29.710498+0000","last_peered":"2024-04-22T08:33:29.710498+0000","last_clean":"2024-04-22T08:33:29.710498+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:29.710498+0000","last_undegraded":"2024-04-22T08:33:29.710498+0000","last_fullsized":"2024-04-22T08:33:29.710498+0000","mapping_epoch":36,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26402816,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13397","reported_seq":13545,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.954959+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:30.954959+0000","last_peered":"2024-04-22T08:33:30.954959+0000","last_clean":"2024-04-22T08:33:30.954959+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:30.954959+0000","last_undegraded":"2024-04-22T08:33:30.954959+0000","last_fullsized":"2024-04-22T08:33:30.954959+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26251264,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10045,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13750","reported_seq":14188,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.762023+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:30.762023+0000","last_peered":"2024-04-22T08:33:30.762023+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:30.762023+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506450,"num_objects":6473,"num_object_clones":0,"num_object_copies":12946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13690,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":281},{"shards":"2","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13600","reported_seq":14437,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.976342+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:30.976342+0000","last_peered":"2024-04-22T08:33:30.976342+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:30.976342+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26977490,"num_objects":6588,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13598,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3195},{"shards":"0,2","objects":268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13446","reported_seq":13960,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.778733+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:30.778733+0000","last_peered":"2024-04-22T08:33:30.778733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:30.778733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":271},{"shards":"2","objects":2915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13490","reported_seq":21168,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.687246+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:30.687246+0000","last_peered":"2024-04-22T08:33:30.687246+0000","last_clean":"2024-04-22T08:33:30.687246+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:30.687246+0000","last_undegraded":"2024-04-22T08:33:30.687246+0000","last_fullsized":"2024-04-22T08:33:30.687246+0000","mapping_epoch":38,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13485","reported_seq":14171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.796257+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:30.796257+0000","last_peered":"2024-04-22T08:33:30.796257+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:30.796257+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27267689,"num_objects":6658,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13484,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3239},{"shards":"0,2","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13426","reported_seq":14050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.903223+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:30.903223+0000","last_peered":"2024-04-22T08:33:30.903223+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:30.903223+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25903104,"num_objects":6324,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2710},{"shards":"0,2","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13478","reported_seq":14037,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.961856+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:30.961856+0000","last_peered":"2024-04-22T08:33:30.961856+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:30.961856+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26870994,"num_objects":6562,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13476,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3097},{"shards":"0,2","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13615","reported_seq":14296,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.702247+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:30.702247+0000","last_peered":"2024-04-22T08:33:30.702247+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:30.702247+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26163620,"num_objects":6391,"num_object_clones":0,"num_object_copies":12782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024593920,"data_stored":1024519675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117074,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635240,"kb_used_data":328800,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93736600,"statfs":{"total":96636764160,"available":95986278400,"internally_reserved":0,"allocated":336691200,"data_stored":336465273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,1,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920836,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647312,"kb_used_data":300680,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93724528,"statfs":{"total":96636764160,"available":95973916672,"internally_reserved":0,"allocated":307896320,"data_stored":307614729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.52100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313302,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755980,"kb_used_data":373316,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615860,"statfs":{"total":96636764160,"available":95862640640,"internally_reserved":0,"allocated":382275584,"data_stored":382060191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.3789999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":4.4210000000000003},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380755968,"data_stored":380752489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307519488,"data_stored":307456273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336318464,"data_stored":336310913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:32.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:33.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:33.758 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-04-22T08:33:32.305268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2038532,"kb_used_data":1002796,"kb_used_omap":12,"kb_used_meta":1035699,"kb_avail":281076988,"statfs":{"total":289910292480,"available":287822835712,"internally_reserved":0,"allocated":1026863104,"data_stored":1026140193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1060555886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,2,3,2,3,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25079808,"num_objects":-6123,"num_object_clones":0,"num_object_copies":-12246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":1544192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005333"},"pg_stats":[{"pgid":"2.f","version":"39'13607","reported_seq":14283,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.969493+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:30.969493+0000","last_peered":"2024-04-22T08:33:30.969493+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:30.969493+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26873856,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13607,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3072},{"shards":"0,2","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13631","reported_seq":13749,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.711100+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:30.711100+0000","last_peered":"2024-04-22T08:33:30.711100+0000","last_clean":"2024-04-22T08:33:30.711100+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:30.711100+0000","last_undegraded":"2024-04-22T08:33:30.711100+0000","last_fullsized":"2024-04-22T08:33:30.711100+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27037696,"num_objects":6601,"num_object_clones":0,"num_object_copies":13202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13410","reported_seq":13909,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.464468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:30.464468+0000","last_peered":"2024-04-22T08:33:30.464468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:30.464468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431512,"num_objects":6454,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":279},{"shards":"2","objects":2885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13665","reported_seq":13698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.798237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:30.798237+0000","last_peered":"2024-04-22T08:33:30.798237+0000","last_clean":"2024-04-22T08:33:30.798237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:30.798237+0000","last_undegraded":"2024-04-22T08:33:30.798237+0000","last_fullsized":"2024-04-22T08:33:30.798237+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26415104,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13665,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13499","reported_seq":13641,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.935797+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:30.935797+0000","last_peered":"2024-04-22T08:33:30.935797+0000","last_clean":"2024-04-22T08:33:30.935797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:30.935797+0000","last_undegraded":"2024-04-22T08:33:30.935797+0000","last_fullsized":"2024-04-22T08:33:30.935797+0000","mapping_epoch":27,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26296320,"num_objects":6420,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13639","reported_seq":17419,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.988075+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:30.988075+0000","last_peered":"2024-04-22T08:33:30.988075+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:30.988075+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":6408,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":6408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":6516736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4334},{"shards":"0,2","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13418","reported_seq":21157,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:29.710498+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:29.710498+0000","last_peered":"2024-04-22T08:33:29.710498+0000","last_clean":"2024-04-22T08:33:29.710498+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:29.710498+0000","last_undegraded":"2024-04-22T08:33:29.710498+0000","last_fullsized":"2024-04-22T08:33:29.710498+0000","mapping_epoch":36,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26402816,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13397","reported_seq":13545,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.954959+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:30.954959+0000","last_peered":"2024-04-22T08:33:30.954959+0000","last_clean":"2024-04-22T08:33:30.954959+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:30.954959+0000","last_undegraded":"2024-04-22T08:33:30.954959+0000","last_fullsized":"2024-04-22T08:33:30.954959+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26251264,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10045,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13750","reported_seq":14188,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.762023+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:30.762023+0000","last_peered":"2024-04-22T08:33:30.762023+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:30.762023+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506450,"num_objects":6473,"num_object_clones":0,"num_object_copies":12946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13690,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":281},{"shards":"2","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13600","reported_seq":14437,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.976342+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:30.976342+0000","last_peered":"2024-04-22T08:33:30.976342+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:30.976342+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26977490,"num_objects":6588,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13598,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3195},{"shards":"0,2","objects":268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13446","reported_seq":13960,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.778733+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:30.778733+0000","last_peered":"2024-04-22T08:33:30.778733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:30.778733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":271},{"shards":"2","objects":2915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13490","reported_seq":21168,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.687246+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:30.687246+0000","last_peered":"2024-04-22T08:33:30.687246+0000","last_clean":"2024-04-22T08:33:30.687246+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:30.687246+0000","last_undegraded":"2024-04-22T08:33:30.687246+0000","last_fullsized":"2024-04-22T08:33:30.687246+0000","mapping_epoch":38,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13485","reported_seq":14171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.796257+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:30.796257+0000","last_peered":"2024-04-22T08:33:30.796257+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:30.796257+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27267689,"num_objects":6658,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13484,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3239},{"shards":"0,2","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13426","reported_seq":14050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.903223+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:30.903223+0000","last_peered":"2024-04-22T08:33:30.903223+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:30.903223+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25903104,"num_objects":6324,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2710},{"shards":"0,2","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13478","reported_seq":14037,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.961856+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:30.961856+0000","last_peered":"2024-04-22T08:33:30.961856+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:30.961856+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26870994,"num_objects":6562,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13476,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3097},{"shards":"0,2","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13615","reported_seq":14296,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.702247+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:30.702247+0000","last_peered":"2024-04-22T08:33:30.702247+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:30.702247+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26163620,"num_objects":6391,"num_object_clones":0,"num_object_copies":12782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024593920,"data_stored":1024519675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117074,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635240,"kb_used_data":328800,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93736600,"statfs":{"total":96636764160,"available":95986278400,"internally_reserved":0,"allocated":336691200,"data_stored":336465273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,1,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920836,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647312,"kb_used_data":300680,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93724528,"statfs":{"total":96636764160,"available":95973916672,"internally_reserved":0,"allocated":307896320,"data_stored":307614729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.52100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313302,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755980,"kb_used_data":373316,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615860,"statfs":{"total":96636764160,"available":95862640640,"internally_reserved":0,"allocated":382275584,"data_stored":382060191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.3789999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":4.4210000000000003},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380755968,"data_stored":380752489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307519488,"data_stored":307456273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336318464,"data_stored":336310913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:33.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:33.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'13607", 'reported_seq': 14283, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.969493+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:30.969493+0000', 'last_peered': '2024-04-22T08:33:30.969493+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:30.969493+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "31'3600", 'ondisk_log_start': "31'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26873856, 'num_objects': 6561, 'num_object_clones': 0, 'num_object_copies': 13122, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3489, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13607, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 305, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3072}, {'shards': '0,2', 'objects': 271}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:33.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13410", 'reported_seq': 13909, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.464468+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:30.464468+0000', 'last_peered': '2024-04-22T08:33:30.464468+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:30.464468+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26431512, 'num_objects': 6454, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3570, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6454, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10237, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 186, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 279}, {'shards': '2', 'objects': 2885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'13639", 'reported_seq': 17419, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.988075+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:30.988075+0000', 'last_peered': '2024-04-22T08:33:30.988075+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:30.988075+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "31'3600", 'ondisk_log_start': "31'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26247168, 'num_objects': 6408, 'num_object_clones': 0, 'num_object_copies': 12816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 260, 'num_objects_unfound': 0, 'num_objects_dirty': 6408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10508, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1808, 'num_bytes_recovered': 6516736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4334}, {'shards': '0,2', 'objects': 260}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13750", 'reported_seq': 14188, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.762023+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:30.762023+0000', 'last_peered': '2024-04-22T08:33:30.762023+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:30.762023+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "31'3700", 'ondisk_log_start': "31'3700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26506450, 'num_objects': 6473, 'num_object_clones': 0, 'num_object_copies': 12946, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3734, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13690, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 161, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 281}, {'shards': '2', 'objects': 2741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'13600", 'reported_seq': 14437, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.976342+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:30.976342+0000', 'last_peered': '2024-04-22T08:33:30.976342+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:30.976342+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "31'3500", 'ondisk_log_start': "31'3500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26977490, 'num_objects': 6588, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3395, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13598, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 381, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3195}, {'shards': '0,2', 'objects': 268}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13446", 'reported_seq': 13960, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.778733+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:30.778733+0000', 'last_peered': '2024-04-22T08:33:30.778733+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:30.778733+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26554368, 'num_objects': 6483, 'num_object_clones': 0, 'num_object_copies': 12966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3570, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9900, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 182, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 271}, {'shards': '2', 'objects': 2915}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:33.760 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'13485", 'reported_seq': 14171, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.796257+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:30.796257+0000', 'last_peered': '2024-04-22T08:33:30.796257+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:30.796257+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27267689, 'num_objects': 6658, 'num_object_clones': 0, 'num_object_copies': 13316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3420, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13484, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 304, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3239}, {'shards': '0,2', 'objects': 308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13426", 'reported_seq': 14050, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.903223+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:30.903223+0000', 'last_peered': '2024-04-22T08:33:30.903223+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:30.903223+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25903104, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 12648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3617, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9657, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 223, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2710}, {'shards': '0,2', 'objects': 290}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'13478", 'reported_seq': 14037, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.961856+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:30.961856+0000', 'last_peered': '2024-04-22T08:33:30.961856+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:30.961856+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "31'3400", 'ondisk_log_start': "31'3400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26870994, 'num_objects': 6562, 'num_object_clones': 0, 'num_object_copies': 13124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3467, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13476, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 242, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3097}, {'shards': '0,2', 'objects': 248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:33.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'13615", 'reported_seq': 14296, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:30.702247+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:30.702247+0000', 'last_peered': '2024-04-22T08:33:30.702247+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:30.702247+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "31'3600", 'ondisk_log_start': "31'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26163620, 'num_objects': 6391, 'num_object_clones': 0, 'num_object_copies': 12782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3617, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13612, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 272, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2777}, {'shards': '0,2', 'objects': 273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:33.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:34.573 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:34.573 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33693/207346 objects degraded (16.250%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":449,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":103673,"data_bytes":424595309,"bytes_used":2087456768,"bytes_avail":287822835712,"bytes_total":289910292480,"degraded_objects":33693,"degraded_total":207346,"degraded_ratio":0.16249650342905095,"misplaced_objects":260,"misplaced_total":207346,"misplaced_ratio":0.0012539426851735746,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":85636,"recovering_keys_per_sec":0,"num_objects_recovered":406,"num_bytes_recovered":1028096,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":344},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:34.607 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:34.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:35.333 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:35.336 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:35.388 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-04-22T08:33:34.305770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2038532,"kb_used_data":1002796,"kb_used_omap":12,"kb_used_meta":1035699,"kb_avail":281076988,"statfs":{"total":289910292480,"available":287822835712,"internally_reserved":0,"allocated":1026863104,"data_stored":1026140193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1060555886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,2,3,2,3,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16949248,"num_objects":-4138,"num_object_clones":0,"num_object_copies":-8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":-252,"num_objects_unfound":0,"num_objects_dirty":-4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":1028096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005363"},"pg_stats":[{"pgid":"2.f","version":"39'13607","reported_seq":14283,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.969493+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:30.969493+0000","last_peered":"2024-04-22T08:33:30.969493+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:30.969493+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26873856,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13607,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3072},{"shards":"0,2","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13631","reported_seq":13749,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.711100+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:30.711100+0000","last_peered":"2024-04-22T08:33:30.711100+0000","last_clean":"2024-04-22T08:33:30.711100+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:30.711100+0000","last_undegraded":"2024-04-22T08:33:30.711100+0000","last_fullsized":"2024-04-22T08:33:30.711100+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27037696,"num_objects":6601,"num_object_clones":0,"num_object_copies":13202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13410","reported_seq":13909,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.464468+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:30.464468+0000","last_peered":"2024-04-22T08:33:30.464468+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:30.464468+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431512,"num_objects":6454,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":279},{"shards":"2","objects":2885}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13665","reported_seq":13698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.798237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:30.798237+0000","last_peered":"2024-04-22T08:33:30.798237+0000","last_clean":"2024-04-22T08:33:30.798237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:30.798237+0000","last_undegraded":"2024-04-22T08:33:30.798237+0000","last_fullsized":"2024-04-22T08:33:30.798237+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26415104,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13665,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13499","reported_seq":13641,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.935797+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:30.935797+0000","last_peered":"2024-04-22T08:33:30.935797+0000","last_clean":"2024-04-22T08:33:30.935797+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:30.935797+0000","last_undegraded":"2024-04-22T08:33:30.935797+0000","last_fullsized":"2024-04-22T08:33:30.935797+0000","mapping_epoch":27,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26296320,"num_objects":6420,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13639","reported_seq":17419,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.988075+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:30.988075+0000","last_peered":"2024-04-22T08:33:30.988075+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:30.988075+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":6408,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":6408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":6516736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4334},{"shards":"0,2","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13418","reported_seq":21157,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:29.710498+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:29.710498+0000","last_peered":"2024-04-22T08:33:29.710498+0000","last_clean":"2024-04-22T08:33:29.710498+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:29.710498+0000","last_undegraded":"2024-04-22T08:33:29.710498+0000","last_fullsized":"2024-04-22T08:33:29.710498+0000","mapping_epoch":36,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26402816,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13397","reported_seq":13545,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.954959+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:30.954959+0000","last_peered":"2024-04-22T08:33:30.954959+0000","last_clean":"2024-04-22T08:33:30.954959+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:30.954959+0000","last_undegraded":"2024-04-22T08:33:30.954959+0000","last_fullsized":"2024-04-22T08:33:30.954959+0000","mapping_epoch":27,"log_start":"28'3300","ondisk_log_start":"28'3300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26251264,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10045,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13750","reported_seq":14188,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.762023+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:30.762023+0000","last_peered":"2024-04-22T08:33:30.762023+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:30.762023+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506450,"num_objects":6473,"num_object_clones":0,"num_object_copies":12946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13690,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":281},{"shards":"2","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13600","reported_seq":14437,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.976342+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:30.976342+0000","last_peered":"2024-04-22T08:33:30.976342+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:30.976342+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26977490,"num_objects":6588,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13598,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3195},{"shards":"0,2","objects":268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13446","reported_seq":13960,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.778733+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:30.778733+0000","last_peered":"2024-04-22T08:33:30.778733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:30.778733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":271},{"shards":"2","objects":2915}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13490","reported_seq":21168,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:30.687246+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:30.687246+0000","last_peered":"2024-04-22T08:33:30.687246+0000","last_clean":"2024-04-22T08:33:30.687246+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:30.687246+0000","last_undegraded":"2024-04-22T08:33:30.687246+0000","last_fullsized":"2024-04-22T08:33:30.687246+0000","mapping_epoch":38,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13485","reported_seq":14171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.796257+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:30.796257+0000","last_peered":"2024-04-22T08:33:30.796257+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:30.796257+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27267689,"num_objects":6658,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13484,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3239},{"shards":"0,2","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13426","reported_seq":14050,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.903223+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:30.903223+0000","last_peered":"2024-04-22T08:33:30.903223+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:30.903223+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25903104,"num_objects":6324,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2710},{"shards":"0,2","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13478","reported_seq":14037,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.961856+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:30.961856+0000","last_peered":"2024-04-22T08:33:30.961856+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:30.961856+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"31'3400","ondisk_log_start":"31'3400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26870994,"num_objects":6562,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13476,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3097},{"shards":"0,2","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13615","reported_seq":14296,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:30.702247+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:30.702247+0000","last_peered":"2024-04-22T08:33:30.702247+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:30.702247+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26163620,"num_objects":6391,"num_object_clones":0,"num_object_copies":12782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":424595309,"num_objects":103673,"num_object_clones":0,"num_object_copies":207346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33693,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":103673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190031,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11654,"num_bytes_recovered":37593705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024593920,"data_stored":1024519675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117074,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635240,"kb_used_data":328800,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93736600,"statfs":{"total":96636764160,"available":95986278400,"internally_reserved":0,"allocated":336691200,"data_stored":336465273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313781967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,1,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920836,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647312,"kb_used_data":300680,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93724528,"statfs":{"total":96636764160,"available":95973916672,"internally_reserved":0,"allocated":307896320,"data_stored":307614729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.52100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313302,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755980,"kb_used_data":373316,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615860,"statfs":{"total":96636764160,"available":95862640640,"internally_reserved":0,"allocated":382275584,"data_stored":382060191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.3789999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":4.4210000000000003},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380755968,"data_stored":380752489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307519488,"data_stored":307456273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336318464,"data_stored":336310913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:38.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:39.186 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:39.188 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:39.232 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-04-22T08:33:38.306760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415870829,"num_objects":101543,"num_object_clones":0,"num_object_copies":203086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192161,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2028688,"kb_used_data":991284,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281086832,"statfs":{"total":289910292480,"available":287832915968,"internally_reserved":0,"allocated":1015074816,"data_stored":1014351905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,6,0,0,5,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17313792,"num_objects":-4227,"num_object_clones":0,"num_object_copies":-8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005492"},"pg_stats":[{"pgid":"2.f","version":"39'13738","reported_seq":14433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.863673+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:35.863673+0000","last_peered":"2024-04-22T08:33:35.863673+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:35.863673+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26337280,"num_objects":6430,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2950},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13771","reported_seq":13889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.693035+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:35.693035+0000","last_peered":"2024-04-22T08:33:35.693035+0000","last_clean":"2024-04-22T08:33:35.693035+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:35.693035+0000","last_undegraded":"2024-04-22T08:33:35.693035+0000","last_fullsized":"2024-04-22T08:33:35.693035+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26464256,"num_objects":6461,"num_object_clones":0,"num_object_copies":12922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13553","reported_seq":14056,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.634435+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:35.634435+0000","last_peered":"2024-04-22T08:33:35.634435+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:35.634435+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25845784,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10380,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":136},{"shards":"2","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13784","reported_seq":13817,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.748599+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:35.748599+0000","last_peered":"2024-04-22T08:33:35.748599+0000","last_clean":"2024-04-22T08:33:35.748599+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:35.748599+0000","last_undegraded":"2024-04-22T08:33:35.748599+0000","last_fullsized":"2024-04-22T08:33:35.748599+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25927680,"num_objects":6330,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13641","reported_seq":13783,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.923465+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:35.923465+0000","last_peered":"2024-04-22T08:33:35.923465+0000","last_clean":"2024-04-22T08:33:35.923465+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:35.923465+0000","last_undegraded":"2024-04-22T08:33:35.923465+0000","last_fullsized":"2024-04-22T08:33:35.923465+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25714688,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13775","reported_seq":17832,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.012424+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:36.012424+0000","last_peered":"2024-04-22T08:33:36.012424+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:36.012424+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1827,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":7081984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13546","reported_seq":21285,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:34.740788+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:34.740788+0000","last_peered":"2024-04-22T08:33:34.740788+0000","last_clean":"2024-04-22T08:33:34.740788+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:34.740788+0000","last_undegraded":"2024-04-22T08:33:34.740788+0000","last_fullsized":"2024-04-22T08:33:34.740788+0000","mapping_epoch":36,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25878528,"num_objects":6318,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9914,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13526","reported_seq":13674,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:36.013573+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:36.013573+0000","last_peered":"2024-04-22T08:33:36.013573+0000","last_clean":"2024-04-22T08:33:36.013573+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:36.013573+0000","last_undegraded":"2024-04-22T08:33:36.013573+0000","last_fullsized":"2024-04-22T08:33:36.013573+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":6280,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13877","reported_seq":14319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.548132+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:35.548132+0000","last_peered":"2024-04-22T08:33:35.548132+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:35.548132+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25986258,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13817,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":154},{"shards":"2","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13722","reported_seq":14575,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.011436+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:36.011436+0000","last_peered":"2024-04-22T08:33:36.011436+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:36.011436+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477778,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3081},{"shards":"0,2","objects":146}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13574","reported_seq":14092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.749814+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:35.749814+0000","last_peered":"2024-04-22T08:33:35.749814+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:35.749814+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":143},{"shards":"2","objects":2789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13629","reported_seq":21307,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.760125+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:35.760125+0000","last_peered":"2024-04-22T08:33:35.760125+0000","last_clean":"2024-04-22T08:33:35.760125+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:35.760125+0000","last_undegraded":"2024-04-22T08:33:35.760125+0000","last_fullsized":"2024-04-22T08:33:35.760125+0000","mapping_epoch":38,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25826514,"num_objects":6307,"num_object_clones":0,"num_object_copies":12614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13627,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13622","reported_seq":14337,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.025520+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:36.025520+0000","last_peered":"2024-04-22T08:33:36.025520+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:36.025520+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26706537,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13621,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3116},{"shards":"0,2","objects":171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13576","reported_seq":14218,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.970259+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:35.970259+0000","last_peered":"2024-04-22T08:33:35.970259+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:35.970259+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25288704,"num_objects":6174,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2569},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13601","reported_seq":14175,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.006561+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:36.006561+0000","last_peered":"2024-04-22T08:33:36.006561+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:36.006561+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26367186,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13599,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2981},{"shards":"0,2","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13751","reported_seq":14448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.965643+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:35.965643+0000","last_peered":"2024-04-22T08:33:35.965643+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:35.965643+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25606564,"num_objects":6255,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2649},{"shards":"0,2","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":415870829,"num_objects":101543,"num_object_clones":0,"num_object_copies":203086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192161,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1012609024,"data_stored":1012534779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653160,"kb_used_data":324576,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93718680,"statfs":{"total":96636764160,"available":95967928320,"internally_reserved":0,"allocated":332365824,"data_stored":332139897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.373}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.316},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920837,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638092,"kb_used_data":299652,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93733748,"statfs":{"total":96636764160,"available":95983357952,"internally_reserved":0,"allocated":306843648,"data_stored":306562057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":346549964},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.17000000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.20300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.22800000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313303,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":367056,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":375865344,"data_stored":375649951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374231040,"data_stored":374227561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306434048,"data_stored":306370833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331943936,"data_stored":331936385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:39.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:40.464 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:40.465 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:40.506 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-22T08:33:40.307489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415321965,"num_objects":101409,"num_object_clones":0,"num_object_copies":202818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192295,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2035932,"kb_used_data":990336,"kb_used_omap":12,"kb_used_meta":1045555,"kb_avail":281079588,"statfs":{"total":289910292480,"available":287825498112,"internally_reserved":0,"allocated":1014104064,"data_stored":1013381153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1070648430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,3,0,0,5,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17862656,"num_objects":-4361,"num_object_clones":0,"num_object_copies":-8722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4361,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"2.f","version":"39'13738","reported_seq":14433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.863673+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:35.863673+0000","last_peered":"2024-04-22T08:33:35.863673+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:35.863673+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26337280,"num_objects":6430,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2950},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13771","reported_seq":13889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.693035+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:35.693035+0000","last_peered":"2024-04-22T08:33:35.693035+0000","last_clean":"2024-04-22T08:33:35.693035+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:35.693035+0000","last_undegraded":"2024-04-22T08:33:35.693035+0000","last_fullsized":"2024-04-22T08:33:35.693035+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26464256,"num_objects":6461,"num_object_clones":0,"num_object_copies":12922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13553","reported_seq":14056,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.634435+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:35.634435+0000","last_peered":"2024-04-22T08:33:35.634435+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:35.634435+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25845784,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10380,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":136},{"shards":"2","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13784","reported_seq":13817,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.748599+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:35.748599+0000","last_peered":"2024-04-22T08:33:35.748599+0000","last_clean":"2024-04-22T08:33:35.748599+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:35.748599+0000","last_undegraded":"2024-04-22T08:33:35.748599+0000","last_fullsized":"2024-04-22T08:33:35.748599+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25927680,"num_objects":6330,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13641","reported_seq":13783,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.923465+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:35.923465+0000","last_peered":"2024-04-22T08:33:35.923465+0000","last_clean":"2024-04-22T08:33:35.923465+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:35.923465+0000","last_undegraded":"2024-04-22T08:33:35.923465+0000","last_fullsized":"2024-04-22T08:33:35.923465+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25714688,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13775","reported_seq":17832,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.012424+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:36.012424+0000","last_peered":"2024-04-22T08:33:36.012424+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:36.012424+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1827,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":7081984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13680","reported_seq":21419,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:39.940365+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:39.940365+0000","last_peered":"2024-04-22T08:33:39.940365+0000","last_clean":"2024-04-22T08:33:39.940365+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:39.940365+0000","last_undegraded":"2024-04-22T08:33:39.940365+0000","last_fullsized":"2024-04-22T08:33:39.940365+0000","mapping_epoch":36,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25329664,"num_objects":6184,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13526","reported_seq":13674,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:36.013573+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:36.013573+0000","last_peered":"2024-04-22T08:33:36.013573+0000","last_clean":"2024-04-22T08:33:36.013573+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:36.013573+0000","last_undegraded":"2024-04-22T08:33:36.013573+0000","last_fullsized":"2024-04-22T08:33:36.013573+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":6280,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13877","reported_seq":14319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.548132+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:35.548132+0000","last_peered":"2024-04-22T08:33:35.548132+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:35.548132+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25986258,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13817,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":154},{"shards":"2","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13722","reported_seq":14575,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.011436+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:36.011436+0000","last_peered":"2024-04-22T08:33:36.011436+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:36.011436+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477778,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3081},{"shards":"0,2","objects":146}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13574","reported_seq":14092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.749814+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:35.749814+0000","last_peered":"2024-04-22T08:33:35.749814+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:35.749814+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":143},{"shards":"2","objects":2789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13629","reported_seq":21307,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.760125+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:35.760125+0000","last_peered":"2024-04-22T08:33:35.760125+0000","last_clean":"2024-04-22T08:33:35.760125+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:35.760125+0000","last_undegraded":"2024-04-22T08:33:35.760125+0000","last_fullsized":"2024-04-22T08:33:35.760125+0000","mapping_epoch":38,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25826514,"num_objects":6307,"num_object_clones":0,"num_object_copies":12614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13627,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13622","reported_seq":14337,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.025520+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:36.025520+0000","last_peered":"2024-04-22T08:33:36.025520+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:36.025520+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26706537,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13621,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3116},{"shards":"0,2","objects":171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13576","reported_seq":14218,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.970259+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:35.970259+0000","last_peered":"2024-04-22T08:33:35.970259+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:35.970259+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25288704,"num_objects":6174,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2569},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13601","reported_seq":14175,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.006561+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:36.006561+0000","last_peered":"2024-04-22T08:33:36.006561+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:36.006561+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26367186,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13599,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2981},{"shards":"0,2","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13751","reported_seq":14448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.965643+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:35.965643+0000","last_peered":"2024-04-22T08:33:35.965643+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:35.965643+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25606564,"num_objects":6255,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2649},{"shards":"0,2","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":415321965,"num_objects":101409,"num_object_clones":0,"num_object_copies":202818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192295,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011625984,"data_stored":1011551739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653160,"kb_used_data":324576,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93718680,"statfs":{"total":96636764160,"available":95967928320,"internally_reserved":0,"allocated":332365824,"data_stored":332139897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.373}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.316},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645336,"kb_used_data":298704,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93726504,"statfs":{"total":96636764160,"available":95975940096,"internally_reserved":0,"allocated":305872896,"data_stored":305591305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313303,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":367056,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":375865344,"data_stored":375649951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374231040,"data_stored":374227561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305451008,"data_stored":305387793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331943936,"data_stored":331936385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'13738", 'reported_seq': 14433, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.863673+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:35.863673+0000', 'last_peered': '2024-04-22T08:33:35.863673+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:35.863673+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "32'3700", 'ondisk_log_start': "32'3700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26337280, 'num_objects': 6430, 'num_object_clones': 0, 'num_object_copies': 12860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3480, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13738, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 314, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2950}, {'shards': '0,2', 'objects': 140}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13553", 'reported_seq': 14056, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.634435+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:35.634435+0000', 'last_peered': '2024-04-22T08:33:35.634435+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:35.634435+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "31'3500", 'ondisk_log_start': "31'3500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25845784, 'num_objects': 6311, 'num_object_clones': 0, 'num_object_copies': 12622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3568, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6311, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10380, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 188, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 136}, {'shards': '2', 'objects': 2744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'13775", 'reported_seq': 17832, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:36.012424+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:36.012424+0000', 'last_peered': '2024-04-22T08:33:36.012424+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:36.012424+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "31'3700", 'ondisk_log_start': "31'3700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25690112, 'num_objects': 6272, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1827, 'num_objects_misplaced': 109, 'num_objects_unfound': 0, 'num_objects_dirty': 6272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10644, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1946, 'num_bytes_recovered': 7081984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4336}, {'shards': '0,2', 'objects': 109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'13877", 'reported_seq': 14319, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.548132+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:35.548132+0000', 'last_peered': '2024-04-22T08:33:35.548132+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:35.548132+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "32'3800", 'ondisk_log_start': "32'3800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25986258, 'num_objects': 6346, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3732, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13817, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 163, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 154}, {'shards': '2', 'objects': 2616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:40.508 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'13722", 'reported_seq': 14575, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:36.011436+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:36.011436+0000', 'last_peered': '2024-04-22T08:33:36.011436+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:36.011436+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "32'3700", 'ondisk_log_start': "32'3700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26477778, 'num_objects': 6466, 'num_object_clones': 0, 'num_object_copies': 12932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3387, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13720, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 389, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3081}, {'shards': '0,2', 'objects': 146}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13574", 'reported_seq': 14092, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.749814+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:35.749814+0000', 'last_peered': '2024-04-22T08:33:35.749814+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:35.749814+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "31'3500", 'ondisk_log_start': "31'3500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26030080, 'num_objects': 6355, 'num_object_clones': 0, 'num_object_copies': 12710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3568, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10028, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 184, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 143}, {'shards': '2', 'objects': 2789}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'13622", 'reported_seq': 14337, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:36.025520+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:36.025520+0000', 'last_peered': '2024-04-22T08:33:36.025520+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:36.025520+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "32'3600", 'ondisk_log_start': "32'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26706537, 'num_objects': 6521, 'num_object_clones': 0, 'num_object_copies': 13042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3406, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13621, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 318, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3116}, {'shards': '0,2', 'objects': 171}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13576", 'reported_seq': 14218, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.970259+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:35.970259+0000', 'last_peered': '2024-04-22T08:33:35.970259+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:35.970259+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "31'3500", 'ondisk_log_start': "31'3500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25288704, 'num_objects': 6174, 'num_object_clones': 0, 'num_object_copies': 12348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3608, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9807, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 232, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2569}, {'shards': '0,2', 'objects': 140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'13601", 'reported_seq': 14175, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:36.006561+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:36.006561+0000', 'last_peered': '2024-04-22T08:33:36.006561+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:36.006561+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "32'3600", 'ondisk_log_start': "32'3600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26367186, 'num_objects': 6439, 'num_object_clones': 0, 'num_object_copies': 12878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3460, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13599, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 249, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2981}, {'shards': '0,2', 'objects': 125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:40.509 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'13751", 'reported_seq': 14448, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:35.965643+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:35.965643+0000', 'last_peered': '2024-04-22T08:33:35.965643+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:35.965643+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "31'3700", 'ondisk_log_start': "31'3700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25606564, 'num_objects': 6255, 'num_object_clones': 0, 'num_object_copies': 12510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3609, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13748, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 280, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2649}, {'shards': '0,2', 'objects': 137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:40.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:41.151 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:41.151 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33645/202818 objects degraded (16.589%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":456,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":101409,"data_bytes":415321965,"bytes_used":2084794368,"bytes_avail":287825498112,"bytes_total":289910292480,"degraded_objects":33645,"degraded_total":202818,"degraded_ratio":0.16588764310859982,"misplaced_objects":109,"misplaced_total":202818,"misplaced_ratio":0.00053742764448914786,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":88040,"recovering_keys_per_sec":0,"num_objects_recovered":404,"num_bytes_recovered":1056768,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":363},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:41.194 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:41.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:41.908 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-22T08:33:40.307489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":415321965,"num_objects":101409,"num_object_clones":0,"num_object_copies":202818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192295,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2035932,"kb_used_data":990336,"kb_used_omap":12,"kb_used_meta":1045555,"kb_avail":281079588,"statfs":{"total":289910292480,"available":287825498112,"internally_reserved":0,"allocated":1014104064,"data_stored":1013381153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1070648430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,3,0,0,5,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17862656,"num_objects":-4361,"num_object_clones":0,"num_object_copies":-8722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4361,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"2.f","version":"39'13738","reported_seq":14433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.863673+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:35.863673+0000","last_peered":"2024-04-22T08:33:35.863673+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:35.863673+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26337280,"num_objects":6430,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2950},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13771","reported_seq":13889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.693035+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:35.693035+0000","last_peered":"2024-04-22T08:33:35.693035+0000","last_clean":"2024-04-22T08:33:35.693035+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:35.693035+0000","last_undegraded":"2024-04-22T08:33:35.693035+0000","last_fullsized":"2024-04-22T08:33:35.693035+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26464256,"num_objects":6461,"num_object_clones":0,"num_object_copies":12922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13553","reported_seq":14056,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.634435+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:35.634435+0000","last_peered":"2024-04-22T08:33:35.634435+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:35.634435+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25845784,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10380,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":136},{"shards":"2","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13784","reported_seq":13817,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.748599+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:35.748599+0000","last_peered":"2024-04-22T08:33:35.748599+0000","last_clean":"2024-04-22T08:33:35.748599+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:35.748599+0000","last_undegraded":"2024-04-22T08:33:35.748599+0000","last_fullsized":"2024-04-22T08:33:35.748599+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25927680,"num_objects":6330,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13641","reported_seq":13783,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.923465+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:35.923465+0000","last_peered":"2024-04-22T08:33:35.923465+0000","last_clean":"2024-04-22T08:33:35.923465+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:35.923465+0000","last_undegraded":"2024-04-22T08:33:35.923465+0000","last_fullsized":"2024-04-22T08:33:35.923465+0000","mapping_epoch":27,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25714688,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13775","reported_seq":17832,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.012424+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:36.012424+0000","last_peered":"2024-04-22T08:33:36.012424+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:36.012424+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1827,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":7081984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13680","reported_seq":21419,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:39.940365+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:39.940365+0000","last_peered":"2024-04-22T08:33:39.940365+0000","last_clean":"2024-04-22T08:33:39.940365+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:39.940365+0000","last_undegraded":"2024-04-22T08:33:39.940365+0000","last_fullsized":"2024-04-22T08:33:39.940365+0000","mapping_epoch":36,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25329664,"num_objects":6184,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13526","reported_seq":13674,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:36.013573+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:36.013573+0000","last_peered":"2024-04-22T08:33:36.013573+0000","last_clean":"2024-04-22T08:33:36.013573+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:36.013573+0000","last_undegraded":"2024-04-22T08:33:36.013573+0000","last_fullsized":"2024-04-22T08:33:36.013573+0000","mapping_epoch":27,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":6280,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'13877","reported_seq":14319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.548132+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:35.548132+0000","last_peered":"2024-04-22T08:33:35.548132+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:35.548132+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25986258,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13817,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":154},{"shards":"2","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13722","reported_seq":14575,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.011436+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:36.011436+0000","last_peered":"2024-04-22T08:33:36.011436+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:36.011436+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477778,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3081},{"shards":"0,2","objects":146}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13574","reported_seq":14092,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.749814+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:35.749814+0000","last_peered":"2024-04-22T08:33:35.749814+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:35.749814+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":143},{"shards":"2","objects":2789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13629","reported_seq":21307,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:35.760125+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:35.760125+0000","last_peered":"2024-04-22T08:33:35.760125+0000","last_clean":"2024-04-22T08:33:35.760125+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:35.760125+0000","last_undegraded":"2024-04-22T08:33:35.760125+0000","last_fullsized":"2024-04-22T08:33:35.760125+0000","mapping_epoch":38,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25826514,"num_objects":6307,"num_object_clones":0,"num_object_copies":12614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13627,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13622","reported_seq":14337,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.025520+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:36.025520+0000","last_peered":"2024-04-22T08:33:36.025520+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:36.025520+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26706537,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13621,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3116},{"shards":"0,2","objects":171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13576","reported_seq":14218,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.970259+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:35.970259+0000","last_peered":"2024-04-22T08:33:35.970259+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:35.970259+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"31'3500","ondisk_log_start":"31'3500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25288704,"num_objects":6174,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2569},{"shards":"0,2","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13601","reported_seq":14175,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:36.006561+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:36.006561+0000","last_peered":"2024-04-22T08:33:36.006561+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:36.006561+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26367186,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13599,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2981},{"shards":"0,2","objects":125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13751","reported_seq":14448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:35.965643+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:35.965643+0000","last_peered":"2024-04-22T08:33:35.965643+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:35.965643+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"31'3700","ondisk_log_start":"31'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25606564,"num_objects":6255,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2649},{"shards":"0,2","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":415321965,"num_objects":101409,"num_object_clones":0,"num_object_copies":202818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":109,"num_objects_unfound":0,"num_objects_dirty":101409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192295,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11853,"num_bytes_recovered":38158953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011625984,"data_stored":1011551739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653160,"kb_used_data":324576,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93718680,"statfs":{"total":96636764160,"available":95967928320,"internally_reserved":0,"allocated":332365824,"data_stored":332139897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.373}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.316},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645336,"kb_used_data":298704,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93726504,"statfs":{"total":96636764160,"available":95975940096,"internally_reserved":0,"allocated":305872896,"data_stored":305591305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313303,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":367056,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":375865344,"data_stored":375649951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374231040,"data_stored":374227561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305451008,"data_stored":305387793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331943936,"data_stored":331936385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:44.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:45.657 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:45.735 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-04-22T08:33:44.308654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407199597,"num_objects":99426,"num_object_clones":0,"num_object_copies":198852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194278,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12092,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2017080,"kb_used_data":979672,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281098440,"statfs":{"total":289910292480,"available":287844802560,"internally_reserved":0,"allocated":1003184128,"data_stored":1002461217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,5,1,2,1,3,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17395712,"num_objects":-4247,"num_object_clones":0,"num_object_copies":-8494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":-260,"num_objects_unfound":0,"num_objects_dirty":-4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003385"},"pg_stats":[{"pgid":"2.f","version":"39'13866","reported_seq":14575,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.857527+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:40.857527+0000","last_peered":"2024-04-22T08:33:40.857527+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:40.857527+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":6302,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13866,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2829},{"shards":"0,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'13913","reported_seq":14031,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:40.749249+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:40.749249+0000","last_peered":"2024-04-22T08:33:40.749249+0000","last_clean":"2024-04-22T08:33:40.749249+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:40.749249+0000","last_undegraded":"2024-04-22T08:33:40.749249+0000","last_fullsized":"2024-04-22T08:33:40.749249+0000","mapping_epoch":27,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25882624,"num_objects":6319,"num_object_clones":0,"num_object_copies":12638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13671","reported_seq":14196,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.784143+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:40.784143+0000","last_peered":"2024-04-22T08:33:40.784143+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:40.784143+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362456,"num_objects":6193,"num_object_clones":0,"num_object_copies":12386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10498,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":18},{"shards":"2","objects":2637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'13935","reported_seq":13968,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:40.703659+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:40.703659+0000","last_peered":"2024-04-22T08:33:40.703659+0000","last_clean":"2024-04-22T08:33:40.703659+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:40.703659+0000","last_undegraded":"2024-04-22T08:33:40.703659+0000","last_fullsized":"2024-04-22T08:33:40.703659+0000","mapping_epoch":27,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25309184,"num_objects":6179,"num_object_clones":0,"num_object_copies":12358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13935,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13762","reported_seq":13904,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:40.969588+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:40.969588+0000","last_peered":"2024-04-22T08:33:40.969588+0000","last_clean":"2024-04-22T08:33:40.969588+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:40.969588+0000","last_undegraded":"2024-04-22T08:33:40.969588+0000","last_fullsized":"2024-04-22T08:33:40.969588+0000","mapping_epoch":27,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'13893","reported_seq":18246,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.967907+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:40.967907+0000","last_peered":"2024-04-22T08:33:40.967907+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:40.967907+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":6154,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13680","reported_seq":21419,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:39.940365+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:39.940365+0000","last_peered":"2024-04-22T08:33:39.940365+0000","last_clean":"2024-04-22T08:33:39.940365+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:39.940365+0000","last_undegraded":"2024-04-22T08:33:39.940365+0000","last_fullsized":"2024-04-22T08:33:39.940365+0000","mapping_epoch":36,"log_start":"31'3600","ondisk_log_start":"31'3600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25329664,"num_objects":6184,"num_object_clones":0,"num_object_copies":12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13650","reported_seq":13798,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:41.012379+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:41.012379+0000","last_peered":"2024-04-22T08:33:41.012379+0000","last_clean":"2024-04-22T08:33:41.012379+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:41.012379+0000","last_undegraded":"2024-04-22T08:33:41.012379+0000","last_fullsized":"2024-04-22T08:33:41.012379+0000","mapping_epoch":27,"log_start":"32'3600","ondisk_log_start":"32'3600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25214976,"num_objects":6156,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14020","reported_seq":14474,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.648906+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:40.648906+0000","last_peered":"2024-04-22T08:33:40.648906+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:40.648906+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400530,"num_objects":6203,"num_object_clones":0,"num_object_copies":12406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13960,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11},{"shards":"2","objects":2479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'13860","reported_seq":14741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:41.001080+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:41.001080+0000","last_peered":"2024-04-22T08:33:41.001080+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:41.001080+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25912530,"num_objects":6328,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13858,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2957},{"shards":"0,2","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13708","reported_seq":14234,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.595047+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:40.595047+0000","last_peered":"2024-04-22T08:33:40.595047+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:40.595047+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25481216,"num_objects":6221,"num_object_clones":0,"num_object_copies":12442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":9},{"shards":"2","objects":2659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13757","reported_seq":21435,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:40.800035+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:40.800035+0000","last_peered":"2024-04-22T08:33:40.800035+0000","last_clean":"2024-04-22T08:33:40.800035+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:40.800035+0000","last_undegraded":"2024-04-22T08:33:40.800035+0000","last_fullsized":"2024-04-22T08:33:40.800035+0000","mapping_epoch":38,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25302226,"num_objects":6179,"num_object_clones":0,"num_object_copies":12358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13755,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13784","reported_seq":14531,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:41.010809+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:41.010809+0000","last_peered":"2024-04-22T08:33:41.010809+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:41.010809+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26042985,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13783,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2970},{"shards":"0,2","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13706","reported_seq":14378,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.922721+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:40.922721+0000","last_peered":"2024-04-22T08:33:40.922721+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:40.922721+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24756224,"num_objects":6044,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9937,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2454},{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13719","reported_seq":14319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.984040+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:40.984040+0000","last_peered":"2024-04-22T08:33:40.984040+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:40.984040+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"32'3700","ondisk_log_start":"32'3700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25883858,"num_objects":6321,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13717,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2876},{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'13879","reported_seq":14586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:40.973224+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:40.973224+0000","last_peered":"2024-04-22T08:33:40.973224+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:40.973224+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25082276,"num_objects":6127,"num_object_clones":0,"num_object_copies":12254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13876,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2526},{"shards":"0,2","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":407199597,"num_objects":99426,"num_object_clones":0,"num_object_copies":198852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194278,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12092,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000787968,"data_stored":1000713723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117076,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636604,"kb_used_data":320300,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93735236,"statfs":{"total":96636764160,"available":95984881664,"internally_reserved":0,"allocated":327987200,"data_stored":327761273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.33200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.20799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645336,"kb_used_data":298704,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93726504,"statfs":{"total":96636764160,"available":95975940096,"internally_reserved":0,"allocated":305872896,"data_stored":305591305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":354938572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313304,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735140,"kb_used_data":360668,"kb_used_omap":4,"kb_used_meta":374459,"kb_avail":93636700,"statfs":{"total":96636764160,"available":95883980800,"internally_reserved":0,"allocated":369324032,"data_stored":369108639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":383446739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,4,1,2,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.221}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":2.2799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367702016,"data_stored":367698537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305451008,"data_stored":305387793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327634944,"data_stored":327627393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:45.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:46.496 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:46.496 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:46.574 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-04-22T08:33:46.309101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397803373,"num_objects":97132,"num_object_clones":0,"num_object_copies":194264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196572,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160697,"ondisk_log_size":160697,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2001008,"kb_used_data":963604,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281114512,"statfs":{"total":289910292480,"available":287861260288,"internally_reserved":0,"allocated":986730496,"data_stored":986007585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,9,13,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26791936,"num_objects":-6541,"num_object_clones":0,"num_object_copies":-13082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-301,"num_objects_misplaced":-260,"num_objects_unfound":0,"num_objects_dirty":-6541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"2.f","version":"39'14035","reported_seq":14754,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.056707+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:46.056707+0000","last_peered":"2024-04-22T08:33:46.056707+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:46.056707+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25120768,"num_objects":6133,"num_object_clones":0,"num_object_copies":12266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14035,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14069","reported_seq":14187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.816522+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:45.816522+0000","last_peered":"2024-04-22T08:33:45.816522+0000","last_clean":"2024-04-22T08:33:45.816522+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:45.816522+0000","last_undegraded":"2024-04-22T08:33:45.816522+0000","last_fullsized":"2024-04-22T08:33:45.816522+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13811","reported_seq":14344,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.743832+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:45.743832+0000","last_peered":"2024-04-22T08:33:45.743832+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:45.743832+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24789016,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10638,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14061","reported_seq":14094,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.706532+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:45.706532+0000","last_peered":"2024-04-22T08:33:45.706532+0000","last_clean":"2024-04-22T08:33:45.706532+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:45.706532+0000","last_undegraded":"2024-04-22T08:33:45.706532+0000","last_fullsized":"2024-04-22T08:33:45.706532+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24793088,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14061,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13885","reported_seq":14027,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.917502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:45.917502+0000","last_peered":"2024-04-22T08:33:45.917502+0000","last_clean":"2024-04-22T08:33:45.917502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:45.917502+0000","last_undegraded":"2024-04-22T08:33:45.917502+0000","last_fullsized":"2024-04-22T08:33:45.917502+0000","mapping_epoch":27,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14052","reported_seq":18609,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.049395+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:46.049395+0000","last_peered":"2024-04-22T08:33:46.049395+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:46.049395+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2196,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13823","reported_seq":21560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:44.985587+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:44.985587+0000","last_peered":"2024-04-22T08:33:44.985587+0000","last_clean":"2024-04-22T08:33:44.985587+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:44.985587+0000","last_undegraded":"2024-04-22T08:33:44.985587+0000","last_fullsized":"2024-04-22T08:33:44.985587+0000","mapping_epoch":36,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13801","reported_seq":13949,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.949626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:45.949626+0000","last_peered":"2024-04-22T08:33:45.949626+0000","last_clean":"2024-04-22T08:33:45.949626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:45.949626+0000","last_undegraded":"2024-04-22T08:33:45.949626+0000","last_fullsized":"2024-04-22T08:33:45.949626+0000","mapping_epoch":27,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24596480,"num_objects":6005,"num_object_clones":0,"num_object_copies":12010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14155","reported_seq":14611,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.793138+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:45.793138+0000","last_peered":"2024-04-22T08:33:45.793138+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:45.793138+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24847570,"num_objects":6068,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14095,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14000","reported_seq":14881,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.022846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:46.022846+0000","last_peered":"2024-04-22T08:33:46.022846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:46.022846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339090,"num_objects":6188,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13854","reported_seq":14386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.697735+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:45.697735+0000","last_peered":"2024-04-22T08:33:45.697735+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:45.697735+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24883200,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13901","reported_seq":21579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.777626+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:45.777626+0000","last_peered":"2024-04-22T08:33:45.777626+0000","last_clean":"2024-04-22T08:33:45.777626+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:45.777626+0000","last_undegraded":"2024-04-22T08:33:45.777626+0000","last_fullsized":"2024-04-22T08:33:45.777626+0000","mapping_epoch":38,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712402,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13899,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13938","reported_seq":14693,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.037069+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:46.037069+0000","last_peered":"2024-04-22T08:33:46.037069+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:46.037069+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25412201,"num_objects":6205,"num_object_clones":0,"num_object_copies":12410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13834","reported_seq":14505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.050618+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:46.050618+0000","last_peered":"2024-04-22T08:33:46.050618+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:46.050618+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":5916,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13858","reported_seq":14458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.934458+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:45.934458+0000","last_peered":"2024-04-22T08:33:45.934458+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:45.934458+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25314514,"num_objects":6182,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13856,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14020","reported_seq":14733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.980977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:45.980977+0000","last_peered":"2024-04-22T08:33:45.980977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:45.980977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24504740,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14017,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":397803373,"num_objects":97132,"num_object_clones":0,"num_object_copies":194264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196572,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983359488,"data_stored":983285243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160697,"ondisk_log_size":160697,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117077,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636196,"kb_used_data":315800,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93735644,"statfs":{"total":96636764160,"available":95985299456,"internally_reserved":0,"allocated":323379200,"data_stored":323153273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.313}]}]},{"osd":1,"up_from":33,"seq":141733920839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628400,"kb_used_data":294056,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93743440,"statfs":{"total":96636764160,"available":95993282560,"internally_reserved":0,"allocated":301113344,"data_stored":300831753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":342355660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.315},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313305,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736412,"kb_used_data":353748,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93635428,"statfs":{"total":96636764160,"available":95882678272,"internally_reserved":0,"allocated":362237952,"data_stored":362022559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360640512,"data_stored":360637033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299728896,"data_stored":299665681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322990080,"data_stored":322982529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14035", 'reported_seq': 14754, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:46.056707+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:46.056707+0000', 'last_peered': '2024-04-22T08:33:46.056707+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:46.056707+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25120768, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 12266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14035, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2665}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13811", 'reported_seq': 14344, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:45.743832+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:45.743832+0000', 'last_peered': '2024-04-22T08:33:45.743832+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:45.743832+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24789016, 'num_objects': 6053, 'num_object_clones': 0, 'num_object_copies': 12106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6053, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10638, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:46.576 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14052", 'reported_seq': 18609, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:46.049395+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:46.049395+0000', 'last_peered': '2024-04-22T08:33:46.049395+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:46.049395+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24555520, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 11990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1686, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10921, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2196, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'14155", 'reported_seq': 14611, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:45.793138+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:45.793138+0000', 'last_peered': '2024-04-22T08:33:45.793138+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:45.793138+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24847570, 'num_objects': 6068, 'num_object_clones': 0, 'num_object_copies': 12136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14095, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14000", 'reported_seq': 14881, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:46.022846+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:46.022846+0000', 'last_peered': '2024-04-22T08:33:46.022846+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:46.022846+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25339090, 'num_objects': 6188, 'num_object_clones': 0, 'num_object_copies': 12376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13998, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2817}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13854", 'reported_seq': 14386, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:45.697735+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:45.697735+0000', 'last_peered': '2024-04-22T08:33:45.697735+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:45.697735+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24883200, 'num_objects': 6075, 'num_object_clones': 0, 'num_object_copies': 12150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10308, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2516}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'13938", 'reported_seq': 14693, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:46.037069+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:46.037069+0000', 'last_peered': '2024-04-22T08:33:46.037069+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:46.037069+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25412201, 'num_objects': 6205, 'num_object_clones': 0, 'num_object_copies': 12410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13937, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13834", 'reported_seq': 14505, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:46.050618+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:46.050618+0000', 'last_peered': '2024-04-22T08:33:46.050618+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:46.050618+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "32'3800", 'ondisk_log_start': "32'3800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24231936, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 11832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10065, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:46.577 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'13858", 'reported_seq': 14458, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:45.934458+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:45.934458+0000', 'last_peered': '2024-04-22T08:33:45.934458+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:45.934458+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25314514, 'num_objects': 6182, 'num_object_clones': 0, 'num_object_copies': 12364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13856, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.578 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:46.578 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14020", 'reported_seq': 14733, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:45.980977+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:45.980977+0000', 'last_peered': '2024-04-22T08:33:45.980977+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:45.980977+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24504740, 'num_objects': 5986, 'num_object_clones': 0, 'num_object_copies': 11972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14017, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:46.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:47.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:47.504 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33392/194264 objects degraded (17.189%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":462,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":97132,"data_bytes":397803373,"bytes_used":2049032192,"bytes_avail":287861260288,"bytes_total":289910292480,"degraded_objects":33392,"degraded_total":194264,"degraded_ratio":0.17188979944817362,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":84285,"recovering_keys_per_sec":0,"num_objects_recovered":561,"num_bytes_recovered":1011712,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":544},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:47.579 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:47.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:48.326 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:48.389 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-04-22T08:33:48.309619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397803373,"num_objects":97132,"num_object_clones":0,"num_object_copies":194264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196572,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160697,"ondisk_log_size":160697,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2001008,"kb_used_data":963604,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281114512,"statfs":{"total":289910292480,"available":287861260288,"internally_reserved":0,"allocated":986730496,"data_stored":986007585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,9,13,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18067456,"num_objects":-4411,"num_object_clones":0,"num_object_copies":-8822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":-109,"num_objects_unfound":0,"num_objects_dirty":-4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":446464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003401"},"pg_stats":[{"pgid":"2.f","version":"39'14035","reported_seq":14754,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.056707+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:46.056707+0000","last_peered":"2024-04-22T08:33:46.056707+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:46.056707+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25120768,"num_objects":6133,"num_object_clones":0,"num_object_copies":12266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14035,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14069","reported_seq":14187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.816522+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:45.816522+0000","last_peered":"2024-04-22T08:33:45.816522+0000","last_clean":"2024-04-22T08:33:45.816522+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:45.816522+0000","last_undegraded":"2024-04-22T08:33:45.816522+0000","last_fullsized":"2024-04-22T08:33:45.816522+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13811","reported_seq":14344,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.743832+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:45.743832+0000","last_peered":"2024-04-22T08:33:45.743832+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:45.743832+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24789016,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10638,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14061","reported_seq":14094,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.706532+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:45.706532+0000","last_peered":"2024-04-22T08:33:45.706532+0000","last_clean":"2024-04-22T08:33:45.706532+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:45.706532+0000","last_undegraded":"2024-04-22T08:33:45.706532+0000","last_fullsized":"2024-04-22T08:33:45.706532+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24793088,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14061,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13885","reported_seq":14027,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.917502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:45.917502+0000","last_peered":"2024-04-22T08:33:45.917502+0000","last_clean":"2024-04-22T08:33:45.917502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:45.917502+0000","last_undegraded":"2024-04-22T08:33:45.917502+0000","last_fullsized":"2024-04-22T08:33:45.917502+0000","mapping_epoch":27,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14052","reported_seq":18609,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.049395+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:46.049395+0000","last_peered":"2024-04-22T08:33:46.049395+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:46.049395+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2196,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13823","reported_seq":21560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:44.985587+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:44.985587+0000","last_peered":"2024-04-22T08:33:44.985587+0000","last_clean":"2024-04-22T08:33:44.985587+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:44.985587+0000","last_undegraded":"2024-04-22T08:33:44.985587+0000","last_fullsized":"2024-04-22T08:33:44.985587+0000","mapping_epoch":36,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13801","reported_seq":13949,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.949626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:45.949626+0000","last_peered":"2024-04-22T08:33:45.949626+0000","last_clean":"2024-04-22T08:33:45.949626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:45.949626+0000","last_undegraded":"2024-04-22T08:33:45.949626+0000","last_fullsized":"2024-04-22T08:33:45.949626+0000","mapping_epoch":27,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24596480,"num_objects":6005,"num_object_clones":0,"num_object_copies":12010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14155","reported_seq":14611,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.793138+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:45.793138+0000","last_peered":"2024-04-22T08:33:45.793138+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:45.793138+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24847570,"num_objects":6068,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14095,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14000","reported_seq":14881,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.022846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:46.022846+0000","last_peered":"2024-04-22T08:33:46.022846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:46.022846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339090,"num_objects":6188,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13854","reported_seq":14386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.697735+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:45.697735+0000","last_peered":"2024-04-22T08:33:45.697735+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:45.697735+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24883200,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13901","reported_seq":21579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.777626+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:45.777626+0000","last_peered":"2024-04-22T08:33:45.777626+0000","last_clean":"2024-04-22T08:33:45.777626+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:45.777626+0000","last_undegraded":"2024-04-22T08:33:45.777626+0000","last_fullsized":"2024-04-22T08:33:45.777626+0000","mapping_epoch":38,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712402,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13899,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13938","reported_seq":14693,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.037069+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:46.037069+0000","last_peered":"2024-04-22T08:33:46.037069+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:46.037069+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25412201,"num_objects":6205,"num_object_clones":0,"num_object_copies":12410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13834","reported_seq":14505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.050618+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:46.050618+0000","last_peered":"2024-04-22T08:33:46.050618+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:46.050618+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":5916,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13858","reported_seq":14458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.934458+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:45.934458+0000","last_peered":"2024-04-22T08:33:45.934458+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:45.934458+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25314514,"num_objects":6182,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13856,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14020","reported_seq":14733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.980977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:45.980977+0000","last_peered":"2024-04-22T08:33:45.980977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:45.980977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24504740,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14017,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":397803373,"num_objects":97132,"num_object_clones":0,"num_object_copies":194264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196572,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983359488,"data_stored":983285243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160697,"ondisk_log_size":160697,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117077,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636196,"kb_used_data":315800,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93735644,"statfs":{"total":96636764160,"available":95985299456,"internally_reserved":0,"allocated":323379200,"data_stored":323153273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.313}]}]},{"osd":1,"up_from":33,"seq":141733920839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628400,"kb_used_data":294056,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93743440,"statfs":{"total":96636764160,"available":95993282560,"internally_reserved":0,"allocated":301113344,"data_stored":300831753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":342355660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.315},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313305,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736412,"kb_used_data":353748,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93635428,"statfs":{"total":96636764160,"available":95882678272,"internally_reserved":0,"allocated":362237952,"data_stored":362022559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360640512,"data_stored":360637033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299728896,"data_stored":299665681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322990080,"data_stored":322982529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:51.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:52.041 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:52.093 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-04-22T08:33:50.313237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397246317,"num_objects":96996,"num_object_clones":0,"num_object_copies":193992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196708,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2002208,"kb_used_data":956612,"kb_used_omap":12,"kb_used_meta":1045555,"kb_avail":281113312,"statfs":{"total":289910292480,"available":287860031488,"internally_reserved":0,"allocated":979570688,"data_stored":978847777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1070648430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,8,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18624512,"num_objects":-4547,"num_object_clones":0,"num_object_copies":-9094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":-109,"num_objects_unfound":0,"num_objects_dirty":-4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":446464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006476"},"pg_stats":[{"pgid":"2.f","version":"39'14035","reported_seq":14754,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.056707+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:46.056707+0000","last_peered":"2024-04-22T08:33:46.056707+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:46.056707+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25120768,"num_objects":6133,"num_object_clones":0,"num_object_copies":12266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14035,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14069","reported_seq":14187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.816522+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:45.816522+0000","last_peered":"2024-04-22T08:33:45.816522+0000","last_clean":"2024-04-22T08:33:45.816522+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:45.816522+0000","last_undegraded":"2024-04-22T08:33:45.816522+0000","last_fullsized":"2024-04-22T08:33:45.816522+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25243648,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13811","reported_seq":14344,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.743832+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:45.743832+0000","last_peered":"2024-04-22T08:33:45.743832+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:45.743832+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24789016,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10638,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14061","reported_seq":14094,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.706532+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:45.706532+0000","last_peered":"2024-04-22T08:33:45.706532+0000","last_clean":"2024-04-22T08:33:45.706532+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:45.706532+0000","last_undegraded":"2024-04-22T08:33:45.706532+0000","last_fullsized":"2024-04-22T08:33:45.706532+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24793088,"num_objects":6053,"num_object_clones":0,"num_object_copies":12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14061,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'13885","reported_seq":14027,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.917502+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:45.917502+0000","last_peered":"2024-04-22T08:33:45.917502+0000","last_clean":"2024-04-22T08:33:45.917502+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:45.917502+0000","last_undegraded":"2024-04-22T08:33:45.917502+0000","last_fullsized":"2024-04-22T08:33:45.917502+0000","mapping_epoch":27,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14052","reported_seq":18609,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.049395+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:46.049395+0000","last_peered":"2024-04-22T08:33:46.049395+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:46.049395+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2196,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13959","reported_seq":21698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:49.967748+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:49.967748+0000","last_peered":"2024-04-22T08:33:49.967748+0000","last_clean":"2024-04-22T08:33:49.967748+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:49.967748+0000","last_undegraded":"2024-04-22T08:33:49.967748+0000","last_fullsized":"2024-04-22T08:33:49.967748+0000","mapping_epoch":36,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24186880,"num_objects":5905,"num_object_clones":0,"num_object_copies":11810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13801","reported_seq":13949,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.949626+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:45.949626+0000","last_peered":"2024-04-22T08:33:45.949626+0000","last_clean":"2024-04-22T08:33:45.949626+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:45.949626+0000","last_undegraded":"2024-04-22T08:33:45.949626+0000","last_fullsized":"2024-04-22T08:33:45.949626+0000","mapping_epoch":27,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24596480,"num_objects":6005,"num_object_clones":0,"num_object_copies":12010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14155","reported_seq":14611,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.793138+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:45.793138+0000","last_peered":"2024-04-22T08:33:45.793138+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:45.793138+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24847570,"num_objects":6068,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14095,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14000","reported_seq":14881,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.022846+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:46.022846+0000","last_peered":"2024-04-22T08:33:46.022846+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:46.022846+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339090,"num_objects":6188,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13854","reported_seq":14386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.697735+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:45.697735+0000","last_peered":"2024-04-22T08:33:45.697735+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:45.697735+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24883200,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'13901","reported_seq":21579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:45.777626+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:45.777626+0000","last_peered":"2024-04-22T08:33:45.777626+0000","last_clean":"2024-04-22T08:33:45.777626+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:45.777626+0000","last_undegraded":"2024-04-22T08:33:45.777626+0000","last_fullsized":"2024-04-22T08:33:45.777626+0000","mapping_epoch":38,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24712402,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13899,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'13938","reported_seq":14693,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.037069+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:46.037069+0000","last_peered":"2024-04-22T08:33:46.037069+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:46.037069+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25412201,"num_objects":6205,"num_object_clones":0,"num_object_copies":12410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13834","reported_seq":14505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:46.050618+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:46.050618+0000","last_peered":"2024-04-22T08:33:46.050618+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:46.050618+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"32'3800","ondisk_log_start":"32'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24231936,"num_objects":5916,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'13858","reported_seq":14458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.934458+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:45.934458+0000","last_peered":"2024-04-22T08:33:45.934458+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:45.934458+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25314514,"num_objects":6182,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13856,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14020","reported_seq":14733,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:45.980977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:45.980977+0000","last_peered":"2024-04-22T08:33:45.980977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:45.980977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24504740,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14017,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":397246317,"num_objects":96996,"num_object_clones":0,"num_object_copies":193992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196708,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12215,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":976154624,"data_stored":976080379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117077,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636196,"kb_used_data":315800,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93735644,"statfs":{"total":96636764160,"available":95985299456,"internally_reserved":0,"allocated":323379200,"data_stored":323153273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.313}]}]},{"osd":1,"up_from":33,"seq":141733920840,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629600,"kb_used_data":287064,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93742240,"statfs":{"total":96636764160,"available":95992053760,"internally_reserved":0,"allocated":293953536,"data_stored":293671945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313305,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736412,"kb_used_data":353748,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93635428,"statfs":{"total":96636764160,"available":95882678272,"internally_reserved":0,"allocated":362237952,"data_stored":362022559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360640512,"data_stored":360637033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292524032,"data_stored":292460817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322990080,"data_stored":322982529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:52.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:52.951 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:52.953 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:52.996 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-04-22T08:33:52.319488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388190061,"num_objects":94785,"num_object_clones":0,"num_object_copies":189570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198919,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12318,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1982544,"kb_used_data":945140,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281132976,"statfs":{"total":289910292480,"available":287880167424,"internally_reserved":0,"allocated":967823360,"data_stored":967100449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,7,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27131904,"num_objects":-6624,"num_object_clones":0,"num_object_copies":-13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-356,"num_objects_misplaced":-109,"num_objects_unfound":0,"num_objects_dirty":-6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":446464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011999"},"pg_stats":[{"pgid":"2.f","version":"39'14175","reported_seq":14894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.069845+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:51.069845+0000","last_peered":"2024-04-22T08:33:51.069845+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:51.069845+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24547328,"num_objects":5993,"num_object_clones":0,"num_object_copies":11986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14175,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14218","reported_seq":14336,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.831347+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:50.831347+0000","last_peered":"2024-04-22T08:33:50.831347+0000","last_clean":"2024-04-22T08:33:50.831347+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:50.831347+0000","last_undegraded":"2024-04-22T08:33:50.831347+0000","last_fullsized":"2024-04-22T08:33:50.831347+0000","mapping_epoch":27,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11274,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13976","reported_seq":14507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.820138+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:50.820138+0000","last_peered":"2024-04-22T08:33:50.820138+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:50.820138+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24113176,"num_objects":5888,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10803,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14197","reported_seq":14230,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.845134+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:50.845134+0000","last_peered":"2024-04-22T08:33:50.845134+0000","last_clean":"2024-04-22T08:33:50.845134+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:50.845134+0000","last_undegraded":"2024-04-22T08:33:50.845134+0000","last_fullsized":"2024-04-22T08:33:50.845134+0000","mapping_epoch":27,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24236032,"num_objects":5917,"num_object_clones":0,"num_object_copies":11834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14197,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14037","reported_seq":14179,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:51.043331+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:51.043331+0000","last_peered":"2024-04-22T08:33:51.043331+0000","last_clean":"2024-04-22T08:33:51.043331+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:51.043331+0000","last_undegraded":"2024-04-22T08:33:51.043331+0000","last_fullsized":"2024-04-22T08:33:51.043331+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24092672,"num_objects":5882,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11129,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14196","reported_seq":18957,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.007113+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:51.007113+0000","last_peered":"2024-04-22T08:33:51.007113+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:51.007113+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23965696,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11065,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13959","reported_seq":21698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:49.967748+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:49.967748+0000","last_peered":"2024-04-22T08:33:49.967748+0000","last_clean":"2024-04-22T08:33:49.967748+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:49.967748+0000","last_undegraded":"2024-04-22T08:33:49.967748+0000","last_fullsized":"2024-04-22T08:33:49.967748+0000","mapping_epoch":36,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24186880,"num_objects":5905,"num_object_clones":0,"num_object_copies":11810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13941","reported_seq":14089,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:51.039701+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:51.039701+0000","last_peered":"2024-04-22T08:33:51.039701+0000","last_clean":"2024-04-22T08:33:51.039701+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:51.039701+0000","last_undegraded":"2024-04-22T08:33:51.039701+0000","last_fullsized":"2024-04-22T08:33:51.039701+0000","mapping_epoch":27,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24023040,"num_objects":5865,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14301","reported_seq":14757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.753786+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:50.753786+0000","last_peered":"2024-04-22T08:33:50.753786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:50.753786+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24249554,"num_objects":5922,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2199}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14138","reported_seq":15019,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.060070+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:51.060070+0000","last_peered":"2024-04-22T08:33:51.060070+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:51.060070+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24773842,"num_objects":6050,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13981","reported_seq":14513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.758554+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:50.758554+0000","last_peered":"2024-04-22T08:33:50.758554+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:50.758554+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24363008,"num_objects":5948,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14066","reported_seq":21743,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.808362+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:50.808362+0000","last_peered":"2024-04-22T08:33:50.808362+0000","last_clean":"2024-04-22T08:33:50.808362+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:50.808362+0000","last_undegraded":"2024-04-22T08:33:50.808362+0000","last_fullsized":"2024-04-22T08:33:50.808362+0000","mapping_epoch":38,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036562,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14064,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14087","reported_seq":14842,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.067543+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:51.067543+0000","last_peered":"2024-04-22T08:33:51.067543+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:51.067543+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24801897,"num_objects":6056,"num_object_clones":0,"num_object_copies":12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13995","reported_seq":14667,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.030139+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:51.030139+0000","last_peered":"2024-04-22T08:33:51.030139+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:51.030139+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23572480,"num_objects":5755,"num_object_clones":0,"num_object_copies":11510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14015","reported_seq":14617,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.005689+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:51.005689+0000","last_peered":"2024-04-22T08:33:51.005689+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:51.005689+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24671442,"num_objects":6025,"num_object_clones":0,"num_object_copies":12050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14162","reported_seq":14875,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.049494+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:51.049494+0000","last_peered":"2024-04-22T08:33:51.049494+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:51.049494+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23923108,"num_objects":5844,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":388190061,"num_objects":94785,"num_object_clones":0,"num_object_copies":189570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198919,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12318,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":964145152,"data_stored":964070907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117078,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635580,"kb_used_data":311092,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93736260,"statfs":{"total":96636764160,"available":95985930240,"internally_reserved":0,"allocated":318558208,"data_stored":318332281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":6.0599999999999996},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.071},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.77300000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920840,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629600,"kb_used_data":287064,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93742240,"statfs":{"total":96636764160,"available":95992053760,"internally_reserved":0,"allocated":293953536,"data_stored":293671945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313306,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717364,"kb_used_data":346984,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93654476,"statfs":{"total":96636764160,"available":95902183424,"internally_reserved":0,"allocated":355311616,"data_stored":355096223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.247},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353419264,"data_stored":353415785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292524032,"data_stored":292460817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318201856,"data_stored":318194305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14175", 'reported_seq': 14894, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.069845+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:51.069845+0000', 'last_peered': '2024-04-22T08:33:51.069845+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:51.069845+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24547328, 'num_objects': 5993, 'num_object_clones': 0, 'num_object_copies': 11986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14175, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2525}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'13976", 'reported_seq': 14507, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:50.820138+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:50.820138+0000', 'last_peered': '2024-04-22T08:33:50.820138+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:50.820138+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24113176, 'num_objects': 5888, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5888, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10803, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14196", 'reported_seq': 18957, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.007113+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:51.007113+0000', 'last_peered': '2024-04-22T08:33:51.007113+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:51.007113+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23965696, 'num_objects': 5851, 'num_object_clones': 0, 'num_object_copies': 11702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1583, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11065, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2299, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'14301", 'reported_seq': 14757, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:50.753786+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:50.753786+0000', 'last_peered': '2024-04-22T08:33:50.753786+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:50.753786+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24249554, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 11844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14241, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2199}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:52.999 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14138", 'reported_seq': 15019, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.060070+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:51.060070+0000', 'last_peered': '2024-04-22T08:33:51.060070+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:51.060070+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24773842, 'num_objects': 6050, 'num_object_clones': 0, 'num_object_copies': 12100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14136, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2679}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:52.999 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:52.999 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'13981", 'reported_seq': 14513, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:50.758554+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:50.758554+0000', 'last_peered': '2024-04-22T08:33:50.758554+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:50.758554+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24363008, 'num_objects': 5948, 'num_object_clones': 0, 'num_object_copies': 11896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10435, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2389}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:52.999 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14087", 'reported_seq': 14842, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.067543+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:51.067543+0000', 'last_peered': '2024-04-22T08:33:51.067543+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:51.067543+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24801897, 'num_objects': 6056, 'num_object_clones': 0, 'num_object_copies': 12112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14086, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'13995", 'reported_seq': 14667, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.030139+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:51.030139+0000', 'last_peered': '2024-04-22T08:33:51.030139+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:51.030139+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23572480, 'num_objects': 5755, 'num_object_clones': 0, 'num_object_copies': 11510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10226, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14015", 'reported_seq': 14617, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.005689+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:51.005689+0000', 'last_peered': '2024-04-22T08:33:51.005689+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:51.005689+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24671442, 'num_objects': 6025, 'num_object_clones': 0, 'num_object_copies': 12050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14013, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:53.002 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14162", 'reported_seq': 14875, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:51.049494+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:51.049494+0000', 'last_peered': '2024-04-22T08:33:51.049494+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:51.049494+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23923108, 'num_objects': 5844, 'num_object_clones': 0, 'num_object_copies': 11688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14159, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:53.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:33:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:53.831 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33289/189570 objects degraded (17.560%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":468,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":94785,"data_bytes":388190061,"bytes_used":2030125056,"bytes_avail":287880167424,"bytes_total":289910292480,"degraded_objects":33289,"degraded_total":189570,"degraded_ratio":0.17560267974890542,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":37168,"recovering_keys_per_sec":0,"num_objects_recovered":465,"num_bytes_recovered":446464,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":551},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:33:53.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:33:53.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:54.903 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:54.903 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:54.976 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-22T08:33:54.319972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388190061,"num_objects":94785,"num_object_clones":0,"num_object_copies":189570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198919,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12318,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1982544,"kb_used_data":945140,"kb_used_omap":12,"kb_used_meta":1037363,"kb_avail":281132976,"statfs":{"total":289910292480,"available":287880167424,"internally_reserved":0,"allocated":967823360,"data_stored":967100449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1062259822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,7,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19009536,"num_objects":-4641,"num_object_clones":0,"num_object_copies":-9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011832"},"pg_stats":[{"pgid":"2.f","version":"39'14175","reported_seq":14894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.069845+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:51.069845+0000","last_peered":"2024-04-22T08:33:51.069845+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:51.069845+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24547328,"num_objects":5993,"num_object_clones":0,"num_object_copies":11986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14175,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14218","reported_seq":14336,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.831347+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:50.831347+0000","last_peered":"2024-04-22T08:33:50.831347+0000","last_clean":"2024-04-22T08:33:50.831347+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:50.831347+0000","last_undegraded":"2024-04-22T08:33:50.831347+0000","last_fullsized":"2024-04-22T08:33:50.831347+0000","mapping_epoch":27,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11274,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'13976","reported_seq":14507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.820138+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:50.820138+0000","last_peered":"2024-04-22T08:33:50.820138+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:50.820138+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24113176,"num_objects":5888,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10803,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14197","reported_seq":14230,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.845134+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:50.845134+0000","last_peered":"2024-04-22T08:33:50.845134+0000","last_clean":"2024-04-22T08:33:50.845134+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:50.845134+0000","last_undegraded":"2024-04-22T08:33:50.845134+0000","last_fullsized":"2024-04-22T08:33:50.845134+0000","mapping_epoch":27,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24236032,"num_objects":5917,"num_object_clones":0,"num_object_copies":11834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14197,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14037","reported_seq":14179,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:51.043331+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:51.043331+0000","last_peered":"2024-04-22T08:33:51.043331+0000","last_clean":"2024-04-22T08:33:51.043331+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:51.043331+0000","last_undegraded":"2024-04-22T08:33:51.043331+0000","last_fullsized":"2024-04-22T08:33:51.043331+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24092672,"num_objects":5882,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11129,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14196","reported_seq":18957,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.007113+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:51.007113+0000","last_peered":"2024-04-22T08:33:51.007113+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:51.007113+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23965696,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11065,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'13959","reported_seq":21698,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:49.967748+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:49.967748+0000","last_peered":"2024-04-22T08:33:49.967748+0000","last_clean":"2024-04-22T08:33:49.967748+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:49.967748+0000","last_undegraded":"2024-04-22T08:33:49.967748+0000","last_fullsized":"2024-04-22T08:33:49.967748+0000","mapping_epoch":36,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24186880,"num_objects":5905,"num_object_clones":0,"num_object_copies":11810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'13941","reported_seq":14089,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:51.039701+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:51.039701+0000","last_peered":"2024-04-22T08:33:51.039701+0000","last_clean":"2024-04-22T08:33:51.039701+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:51.039701+0000","last_undegraded":"2024-04-22T08:33:51.039701+0000","last_fullsized":"2024-04-22T08:33:51.039701+0000","mapping_epoch":27,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24023040,"num_objects":5865,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133154+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:18.133154+0000","last_peered":"2024-04-22T08:32:18.133154+0000","last_clean":"2024-04-22T08:32:18.133154+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:18.133154+0000","last_undegraded":"2024-04-22T08:32:18.133154+0000","last_fullsized":"2024-04-22T08:32:18.133154+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14301","reported_seq":14757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.753786+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:50.753786+0000","last_peered":"2024-04-22T08:33:50.753786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:50.753786+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24249554,"num_objects":5922,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2199}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14138","reported_seq":15019,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.060070+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:51.060070+0000","last_peered":"2024-04-22T08:33:51.060070+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:51.060070+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24773842,"num_objects":6050,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'13981","reported_seq":14513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:50.758554+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:50.758554+0000","last_peered":"2024-04-22T08:33:50.758554+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:50.758554+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24363008,"num_objects":5948,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14066","reported_seq":21743,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:50.808362+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:50.808362+0000","last_peered":"2024-04-22T08:33:50.808362+0000","last_clean":"2024-04-22T08:33:50.808362+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:50.808362+0000","last_undegraded":"2024-04-22T08:33:50.808362+0000","last_fullsized":"2024-04-22T08:33:50.808362+0000","mapping_epoch":38,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036562,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14064,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14087","reported_seq":14842,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.067543+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:51.067543+0000","last_peered":"2024-04-22T08:33:51.067543+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:51.067543+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24801897,"num_objects":6056,"num_object_clones":0,"num_object_copies":12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'13995","reported_seq":14667,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.030139+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:51.030139+0000","last_peered":"2024-04-22T08:33:51.030139+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:51.030139+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23572480,"num_objects":5755,"num_object_clones":0,"num_object_copies":11510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14015","reported_seq":14617,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.005689+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:51.005689+0000","last_peered":"2024-04-22T08:33:51.005689+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:51.005689+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24671442,"num_objects":6025,"num_object_clones":0,"num_object_copies":12050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14162","reported_seq":14875,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:51.049494+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:51.049494+0000","last_peered":"2024-04-22T08:33:51.049494+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:51.049494+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23923108,"num_objects":5844,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":388190061,"num_objects":94785,"num_object_clones":0,"num_object_copies":189570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198919,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12318,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":964145152,"data_stored":964070907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117078,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635580,"kb_used_data":311092,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93736260,"statfs":{"total":96636764160,"available":95985930240,"internally_reserved":0,"allocated":318558208,"data_stored":318332281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":6.0599999999999996},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.071},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.77300000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920840,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629600,"kb_used_data":287064,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93742240,"statfs":{"total":96636764160,"available":95992053760,"internally_reserved":0,"allocated":293953536,"data_stored":293671945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":350744268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313306,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717364,"kb_used_data":346984,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93654476,"statfs":{"total":96636764160,"available":95902183424,"internally_reserved":0,"allocated":355311616,"data_stored":355096223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":379252435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.247},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353419264,"data_stored":353415785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292524032,"data_stored":292460817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318201856,"data_stored":318194305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:57.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:58.688 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:58.691 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-04-22T08:33:58.320905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":378728301,"num_objects":92475,"num_object_clones":0,"num_object_copies":184950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201229,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005568,"kb_used_data":925404,"kb_used_omap":12,"kb_used_meta":1080115,"kb_avail":281109952,"statfs":{"total":289910292480,"available":287856590848,"internally_reserved":0,"allocated":947613696,"data_stored":946890785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1106037870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,18,3,4,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19075072,"num_objects":-4657,"num_object_clones":0,"num_object_copies":-9314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011803"},"pg_stats":[{"pgid":"2.f","version":"39'14325","reported_seq":15044,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.106421+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:56.106421+0000","last_peered":"2024-04-22T08:33:56.106421+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:56.106421+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23932928,"num_objects":5843,"num_object_clones":0,"num_object_copies":11686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14325,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2375}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14355","reported_seq":14473,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.705219+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:55.705219+0000","last_peered":"2024-04-22T08:33:55.705219+0000","last_clean":"2024-04-22T08:33:55.705219+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:55.705219+0000","last_undegraded":"2024-04-22T08:33:55.705219+0000","last_fullsized":"2024-04-22T08:33:55.705219+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24072192,"num_objects":5877,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14136","reported_seq":14666,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.785762+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:55.785762+0000","last_peered":"2024-04-22T08:33:55.785762+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:55.785762+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457816,"num_objects":5728,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10963,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14357","reported_seq":14391,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.835809+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:55.835809+0000","last_peered":"2024-04-22T08:33:55.835809+0000","last_clean":"2024-04-22T08:33:55.835809+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:55.835809+0000","last_undegraded":"2024-04-22T08:33:55.835809+0000","last_fullsized":"2024-04-22T08:33:55.835809+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23580672,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14189","reported_seq":14331,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:56.039010+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:56.039010+0000","last_peered":"2024-04-22T08:33:56.039010+0000","last_clean":"2024-04-22T08:33:56.039010+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:56.039010+0000","last_undegraded":"2024-04-22T08:33:56.039010+0000","last_fullsized":"2024-04-22T08:33:56.039010+0000","mapping_epoch":27,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":5730,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14336","reported_seq":19297,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.109205+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:56.109205+0000","last_peered":"2024-04-22T08:33:56.109205+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:56.109205+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23392256,"num_objects":5711,"num_object_clones":0,"num_object_copies":11422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2398,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14102","reported_seq":21840,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.096919+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:55.096919+0000","last_peered":"2024-04-22T08:33:55.096919+0000","last_clean":"2024-04-22T08:33:55.096919+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:55.096919+0000","last_undegraded":"2024-04-22T08:33:55.096919+0000","last_fullsized":"2024-04-22T08:33:55.096919+0000","mapping_epoch":36,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23601152,"num_objects":5762,"num_object_clones":0,"num_object_copies":11524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14057","reported_seq":14205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.996564+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:55.996564+0000","last_peered":"2024-04-22T08:33:55.996564+0000","last_clean":"2024-04-22T08:33:55.996564+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:55.996564+0000","last_undegraded":"2024-04-22T08:33:55.996564+0000","last_fullsized":"2024-04-22T08:33:55.996564+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23547904,"num_objects":5749,"num_object_clones":0,"num_object_copies":11498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14436","reported_seq":14892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.830171+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:55.830171+0000","last_peered":"2024-04-22T08:33:55.830171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:55.830171+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696594,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14376,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14291","reported_seq":15172,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.071530+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:56.071530+0000","last_peered":"2024-04-22T08:33:56.071530+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:56.071530+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24147154,"num_objects":5897,"num_object_clones":0,"num_object_copies":11794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14116","reported_seq":14648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.859914+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:55.859914+0000","last_peered":"2024-04-22T08:33:55.859914+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:55.859914+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23810048,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14211","reported_seq":21889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.808281+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:55.808281+0000","last_peered":"2024-04-22T08:33:55.808281+0000","last_clean":"2024-04-22T08:33:55.808281+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:55.808281+0000","last_undegraded":"2024-04-22T08:33:55.808281+0000","last_fullsized":"2024-04-22T08:33:55.808281+0000","mapping_epoch":38,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442642,"num_objects":5725,"num_object_clones":0,"num_object_copies":11450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14237","reported_seq":14992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.082512+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:56.082512+0000","last_peered":"2024-04-22T08:33:56.082512+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:56.082512+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24187497,"num_objects":5906,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14236,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14123","reported_seq":14795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.113062+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:56.113062+0000","last_peered":"2024-04-22T08:33:56.113062+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:56.113062+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23048192,"num_objects":5627,"num_object_clones":0,"num_object_copies":11254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14170","reported_seq":14771,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.038114+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:56.038114+0000","last_peered":"2024-04-22T08:33:56.038114+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:56.038114+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036562,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14168,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14313","reported_seq":15026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.052921+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:56.052921+0000","last_peered":"2024-04-22T08:33:56.052921+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:56.052921+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23304612,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14310,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":378728301,"num_objects":92475,"num_object_clones":0,"num_object_copies":184950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201229,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945246208,"data_stored":945171963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117079,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635176,"kb_used_data":306592,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93736664,"statfs":{"total":96636764160,"available":95986343936,"internally_reserved":0,"allocated":313950208,"data_stored":313724281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920841,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653216,"kb_used_data":280208,"kb_used_omap":4,"kb_used_meta":372987,"kb_avail":93718624,"statfs":{"total":96636764160,"available":95967870976,"internally_reserved":0,"allocated":286932992,"data_stored":286651401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":381939404},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313307,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717176,"kb_used_data":338604,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93654664,"statfs":{"total":96636764160,"available":95902375936,"internally_reserved":0,"allocated":346730496,"data_stored":346515103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":387641043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,15,0,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346329088,"data_stored":346325609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285413376,"data_stored":285350161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:58.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:33:59.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:33:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:33:59.527 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-04-22T08:33:58.320905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":378728301,"num_objects":92475,"num_object_clones":0,"num_object_copies":184950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201229,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005568,"kb_used_data":925404,"kb_used_omap":12,"kb_used_meta":1080115,"kb_avail":281109952,"statfs":{"total":289910292480,"available":287856590848,"internally_reserved":0,"allocated":947613696,"data_stored":946890785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1106037870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,18,3,4,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19075072,"num_objects":-4657,"num_object_clones":0,"num_object_copies":-9314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011803"},"pg_stats":[{"pgid":"2.f","version":"39'14325","reported_seq":15044,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.106421+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:56.106421+0000","last_peered":"2024-04-22T08:33:56.106421+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:56.106421+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23932928,"num_objects":5843,"num_object_clones":0,"num_object_copies":11686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14325,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2375}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14355","reported_seq":14473,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.705219+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:55.705219+0000","last_peered":"2024-04-22T08:33:55.705219+0000","last_clean":"2024-04-22T08:33:55.705219+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:55.705219+0000","last_undegraded":"2024-04-22T08:33:55.705219+0000","last_fullsized":"2024-04-22T08:33:55.705219+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24072192,"num_objects":5877,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14136","reported_seq":14666,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.785762+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:55.785762+0000","last_peered":"2024-04-22T08:33:55.785762+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:55.785762+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457816,"num_objects":5728,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10963,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14357","reported_seq":14391,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.835809+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:55.835809+0000","last_peered":"2024-04-22T08:33:55.835809+0000","last_clean":"2024-04-22T08:33:55.835809+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:55.835809+0000","last_undegraded":"2024-04-22T08:33:55.835809+0000","last_fullsized":"2024-04-22T08:33:55.835809+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23580672,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14189","reported_seq":14331,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:56.039010+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:56.039010+0000","last_peered":"2024-04-22T08:33:56.039010+0000","last_clean":"2024-04-22T08:33:56.039010+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:56.039010+0000","last_undegraded":"2024-04-22T08:33:56.039010+0000","last_fullsized":"2024-04-22T08:33:56.039010+0000","mapping_epoch":27,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":5730,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14336","reported_seq":19297,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.109205+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:56.109205+0000","last_peered":"2024-04-22T08:33:56.109205+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:56.109205+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23392256,"num_objects":5711,"num_object_clones":0,"num_object_copies":11422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2398,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14102","reported_seq":21840,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.096919+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:33:55.096919+0000","last_peered":"2024-04-22T08:33:55.096919+0000","last_clean":"2024-04-22T08:33:55.096919+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:33:55.096919+0000","last_undegraded":"2024-04-22T08:33:55.096919+0000","last_fullsized":"2024-04-22T08:33:55.096919+0000","mapping_epoch":36,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23601152,"num_objects":5762,"num_object_clones":0,"num_object_copies":11524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14057","reported_seq":14205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.996564+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:55.996564+0000","last_peered":"2024-04-22T08:33:55.996564+0000","last_clean":"2024-04-22T08:33:55.996564+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:55.996564+0000","last_undegraded":"2024-04-22T08:33:55.996564+0000","last_fullsized":"2024-04-22T08:33:55.996564+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23547904,"num_objects":5749,"num_object_clones":0,"num_object_copies":11498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14436","reported_seq":14892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.830171+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:55.830171+0000","last_peered":"2024-04-22T08:33:55.830171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:55.830171+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696594,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14376,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14291","reported_seq":15172,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.071530+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:56.071530+0000","last_peered":"2024-04-22T08:33:56.071530+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:56.071530+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24147154,"num_objects":5897,"num_object_clones":0,"num_object_copies":11794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14116","reported_seq":14648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.859914+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:55.859914+0000","last_peered":"2024-04-22T08:33:55.859914+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:55.859914+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23810048,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14211","reported_seq":21889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.808281+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:55.808281+0000","last_peered":"2024-04-22T08:33:55.808281+0000","last_clean":"2024-04-22T08:33:55.808281+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:55.808281+0000","last_undegraded":"2024-04-22T08:33:55.808281+0000","last_fullsized":"2024-04-22T08:33:55.808281+0000","mapping_epoch":38,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442642,"num_objects":5725,"num_object_clones":0,"num_object_copies":11450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14237","reported_seq":14992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.082512+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:56.082512+0000","last_peered":"2024-04-22T08:33:56.082512+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:56.082512+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24187497,"num_objects":5906,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14236,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14123","reported_seq":14795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.113062+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:56.113062+0000","last_peered":"2024-04-22T08:33:56.113062+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:56.113062+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23048192,"num_objects":5627,"num_object_clones":0,"num_object_copies":11254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14170","reported_seq":14771,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.038114+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:56.038114+0000","last_peered":"2024-04-22T08:33:56.038114+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:56.038114+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036562,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14168,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14313","reported_seq":15026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.052921+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:56.052921+0000","last_peered":"2024-04-22T08:33:56.052921+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:56.052921+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23304612,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14310,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":378728301,"num_objects":92475,"num_object_clones":0,"num_object_copies":184950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201229,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945246208,"data_stored":945171963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117079,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635176,"kb_used_data":306592,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93736664,"statfs":{"total":96636764160,"available":95986343936,"internally_reserved":0,"allocated":313950208,"data_stored":313724281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920841,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653216,"kb_used_data":280208,"kb_used_omap":4,"kb_used_meta":372987,"kb_avail":93718624,"statfs":{"total":96636764160,"available":95967870976,"internally_reserved":0,"allocated":286932992,"data_stored":286651401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":381939404},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313307,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717176,"kb_used_data":338604,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93654664,"statfs":{"total":96636764160,"available":95902375936,"internally_reserved":0,"allocated":346730496,"data_stored":346515103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":387641043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,15,0,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346329088,"data_stored":346325609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":285413376,"data_stored":285350161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14325", 'reported_seq': 15044, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.106421+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:33:56.106421+0000', 'last_peered': '2024-04-22T08:33:56.106421+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:33:56.106421+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23932928, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 11686, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14325, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2375}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'14136", 'reported_seq': 14666, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:55.785762+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:33:55.785762+0000', 'last_peered': '2024-04-22T08:33:55.785762+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:33:55.785762+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23457816, 'num_objects': 5728, 'num_object_clones': 0, 'num_object_copies': 11456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5728, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10963, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14336", 'reported_seq': 19297, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.109205+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:33:56.109205+0000', 'last_peered': '2024-04-22T08:33:56.109205+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:33:56.109205+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23392256, 'num_objects': 5711, 'num_object_clones': 0, 'num_object_copies': 11422, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1484, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11205, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2398, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:33:59.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'14436", 'reported_seq': 14892, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:55.830171+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:33:55.830171+0000', 'last_peered': '2024-04-22T08:33:55.830171+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:33:55.830171+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'4400", 'ondisk_log_start': "35'4400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23696594, 'num_objects': 5787, 'num_object_clones': 0, 'num_object_copies': 11574, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14376, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14291", 'reported_seq': 15172, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.071530+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:33:56.071530+0000', 'last_peered': '2024-04-22T08:33:56.071530+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:33:56.071530+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'4200", 'ondisk_log_start': "35'4200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24147154, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 11794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14289, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2526}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'14116", 'reported_seq': 14648, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:55.859914+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:33:55.859914+0000', 'last_peered': '2024-04-22T08:33:55.859914+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:33:55.859914+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23810048, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 11626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10570, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2254}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14237", 'reported_seq': 14992, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.082512+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:33:56.082512+0000', 'last_peered': '2024-04-22T08:33:56.082512+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:33:56.082512+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4200", 'ondisk_log_start': "35'4200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24187497, 'num_objects': 5906, 'num_object_clones': 0, 'num_object_copies': 11812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14236, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14123", 'reported_seq': 14795, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.113062+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:33:56.113062+0000', 'last_peered': '2024-04-22T08:33:56.113062+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:33:56.113062+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23048192, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 11254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10354, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:33:59.530 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14170", 'reported_seq': 14771, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.038114+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:33:56.038114+0000', 'last_peered': '2024-04-22T08:33:56.038114+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:33:56.038114+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24036562, 'num_objects': 5870, 'num_object_clones': 0, 'num_object_copies': 11740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14168, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.531 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:33:59.531 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14313", 'reported_seq': 15026, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:33:56.052921+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:33:56.052921+0000', 'last_peered': '2024-04-22T08:33:56.052921+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:33:56.052921+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23304612, 'num_objects': 5693, 'num_object_clones': 0, 'num_object_copies': 11386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14310, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2095}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:33:59.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:00.515 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:00.515 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33190/184634 objects degraded (17.976%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":92317,"data_bytes":378081133,"bytes_used":2029633536,"bytes_avail":287880658944,"bytes_total":289910292480,"degraded_objects":33190,"degraded_total":184634,"degraded_ratio":0.1797610407617232,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":400},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:00.575 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:00.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:01.418 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:01.531 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-04-22T08:34:00.321336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":378081133,"num_objects":92317,"num_object_clones":0,"num_object_copies":184634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201387,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1982064,"kb_used_data":918420,"kb_used_omap":12,"kb_used_meta":1063603,"kb_avail":281133456,"statfs":{"total":289910292480,"available":287880658944,"internally_reserved":0,"allocated":940462080,"data_stored":939739169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1089129582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,1,1,23,0,3,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19722240,"num_objects":-4815,"num_object_clones":0,"num_object_copies":-9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4815,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011716"},"pg_stats":[{"pgid":"2.f","version":"39'14325","reported_seq":15044,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.106421+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:33:56.106421+0000","last_peered":"2024-04-22T08:33:56.106421+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:33:56.106421+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23932928,"num_objects":5843,"num_object_clones":0,"num_object_copies":11686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14325,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2375}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14355","reported_seq":14473,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.705219+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:33:55.705219+0000","last_peered":"2024-04-22T08:33:55.705219+0000","last_clean":"2024-04-22T08:33:55.705219+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:33:55.705219+0000","last_undegraded":"2024-04-22T08:33:55.705219+0000","last_fullsized":"2024-04-22T08:33:55.705219+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24072192,"num_objects":5877,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14136","reported_seq":14666,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.785762+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:33:55.785762+0000","last_peered":"2024-04-22T08:33:55.785762+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:33:55.785762+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23457816,"num_objects":5728,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10963,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14357","reported_seq":14391,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.835809+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:33:55.835809+0000","last_peered":"2024-04-22T08:33:55.835809+0000","last_clean":"2024-04-22T08:33:55.835809+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:33:55.835809+0000","last_undegraded":"2024-04-22T08:33:55.835809+0000","last_fullsized":"2024-04-22T08:33:55.835809+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23580672,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14189","reported_seq":14331,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:56.039010+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:33:56.039010+0000","last_peered":"2024-04-22T08:33:56.039010+0000","last_clean":"2024-04-22T08:33:56.039010+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:33:56.039010+0000","last_undegraded":"2024-04-22T08:33:56.039010+0000","last_fullsized":"2024-04-22T08:33:56.039010+0000","mapping_epoch":27,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":5730,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14336","reported_seq":19297,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.109205+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:33:56.109205+0000","last_peered":"2024-04-22T08:33:56.109205+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:33:56.109205+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23392256,"num_objects":5711,"num_object_clones":0,"num_object_copies":11422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2398,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14260","reported_seq":21999,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.076814+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:00.076814+0000","last_peered":"2024-04-22T08:34:00.076814+0000","last_clean":"2024-04-22T08:34:00.076814+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:00.076814+0000","last_undegraded":"2024-04-22T08:34:00.076814+0000","last_fullsized":"2024-04-22T08:34:00.076814+0000","mapping_epoch":36,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":5604,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14057","reported_seq":14205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.996564+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:33:55.996564+0000","last_peered":"2024-04-22T08:33:55.996564+0000","last_clean":"2024-04-22T08:33:55.996564+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:33:55.996564+0000","last_undegraded":"2024-04-22T08:33:55.996564+0000","last_fullsized":"2024-04-22T08:33:55.996564+0000","mapping_epoch":27,"log_start":"35'4000","ondisk_log_start":"35'4000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23547904,"num_objects":5749,"num_object_clones":0,"num_object_copies":11498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14436","reported_seq":14892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.830171+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:33:55.830171+0000","last_peered":"2024-04-22T08:33:55.830171+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:33:55.830171+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696594,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14376,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14291","reported_seq":15172,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.071530+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:33:56.071530+0000","last_peered":"2024-04-22T08:33:56.071530+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:33:56.071530+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24147154,"num_objects":5897,"num_object_clones":0,"num_object_copies":11794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14116","reported_seq":14648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:55.859914+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:33:55.859914+0000","last_peered":"2024-04-22T08:33:55.859914+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:33:55.859914+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23810048,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14211","reported_seq":21889,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:55.808281+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:33:55.808281+0000","last_peered":"2024-04-22T08:33:55.808281+0000","last_clean":"2024-04-22T08:33:55.808281+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:33:55.808281+0000","last_undegraded":"2024-04-22T08:33:55.808281+0000","last_fullsized":"2024-04-22T08:33:55.808281+0000","mapping_epoch":38,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442642,"num_objects":5725,"num_object_clones":0,"num_object_copies":11450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14237","reported_seq":14992,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.082512+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:33:56.082512+0000","last_peered":"2024-04-22T08:33:56.082512+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:33:56.082512+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24187497,"num_objects":5906,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14236,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14123","reported_seq":14795,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.113062+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:33:56.113062+0000","last_peered":"2024-04-22T08:33:56.113062+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:33:56.113062+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23048192,"num_objects":5627,"num_object_clones":0,"num_object_copies":11254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14170","reported_seq":14771,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.038114+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:33:56.038114+0000","last_peered":"2024-04-22T08:33:56.038114+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:33:56.038114+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4100","ondisk_log_start":"35'4100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036562,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14168,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14313","reported_seq":15026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:33:56.052921+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:33:56.052921+0000","last_peered":"2024-04-22T08:33:56.052921+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:33:56.052921+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23304612,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14310,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":378081133,"num_objects":92317,"num_object_clones":0,"num_object_copies":184634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201387,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12417,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":937930752,"data_stored":937856507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117079,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635176,"kb_used_data":306592,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93736664,"statfs":{"total":96636764160,"available":95986343936,"internally_reserved":0,"allocated":313950208,"data_stored":313724281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629712,"kb_used_data":273224,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93742128,"statfs":{"total":96636764160,"available":95991939072,"internally_reserved":0,"allocated":279781376,"data_stored":279499785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36099999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313307,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717176,"kb_used_data":338604,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93654664,"statfs":{"total":96636764160,"available":95902375936,"internally_reserved":0,"allocated":346730496,"data_stored":346515103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":387641043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,15,0,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.3009999999999999,"15min":0.64500000000000002},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":7.9619999999999997,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1819999999999999,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3360000000000003,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346329088,"data_stored":346325609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278097920,"data_stored":278034705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:04.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:05.368 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:05.369 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:05.410 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-04-22T08:34:04.322308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":369164141,"num_objects":90140,"num_object_clones":0,"num_object_copies":180280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203564,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12512,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1963404,"kb_used_data":907944,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281152116,"statfs":{"total":289910292480,"available":287899766784,"internally_reserved":0,"allocated":929734656,"data_stored":929011745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1080740974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,17,3,9,1,6,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19025920,"num_objects":-4645,"num_object_clones":0,"num_object_copies":-9290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002820"},"pg_stats":[{"pgid":"2.f","version":"39'14455","reported_seq":15173,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.092080+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:01.092080+0000","last_peered":"2024-04-22T08:34:01.092080+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:01.092080+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23400448,"num_objects":5713,"num_object_clones":0,"num_object_copies":11426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14455,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2245}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14510","reported_seq":14628,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.898213+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:00.898213+0000","last_peered":"2024-04-22T08:34:00.898213+0000","last_clean":"2024-04-22T08:34:00.898213+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:00.898213+0000","last_undegraded":"2024-04-22T08:34:00.898213+0000","last_fullsized":"2024-04-22T08:34:00.898213+0000","mapping_epoch":27,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23437312,"num_objects":5722,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14284","reported_seq":14818,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.877653+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:00.877653+0000","last_peered":"2024-04-22T08:34:00.877653+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:00.877653+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851608,"num_objects":5580,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11111,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14493","reported_seq":14527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.848512+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:00.848512+0000","last_peered":"2024-04-22T08:34:00.848512+0000","last_clean":"2024-04-22T08:34:00.848512+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:00.848512+0000","last_undegraded":"2024-04-22T08:34:00.848512+0000","last_fullsized":"2024-04-22T08:34:00.848512+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23023616,"num_objects":5621,"num_object_clones":0,"num_object_copies":11242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14493,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14340","reported_seq":14482,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:01.116113+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:01.116113+0000","last_peered":"2024-04-22T08:34:01.116113+0000","last_clean":"2024-04-22T08:34:01.116113+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:01.116113+0000","last_undegraded":"2024-04-22T08:34:01.116113+0000","last_fullsized":"2024-04-22T08:34:01.116113+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851584,"num_objects":5579,"num_object_clones":0,"num_object_copies":11158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14502","reported_seq":19653,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.142647+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:01.142647+0000","last_peered":"2024-04-22T08:34:01.142647+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:01.142647+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22712320,"num_objects":5545,"num_object_clones":0,"num_object_copies":11090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14260","reported_seq":21999,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.076814+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:00.076814+0000","last_peered":"2024-04-22T08:34:00.076814+0000","last_clean":"2024-04-22T08:34:00.076814+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:00.076814+0000","last_undegraded":"2024-04-22T08:34:00.076814+0000","last_fullsized":"2024-04-22T08:34:00.076814+0000","mapping_epoch":36,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":5604,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14212","reported_seq":14360,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:01.081396+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:01.081396+0000","last_peered":"2024-04-22T08:34:01.081396+0000","last_clean":"2024-04-22T08:34:01.081396+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:01.081396+0000","last_undegraded":"2024-04-22T08:34:01.081396+0000","last_fullsized":"2024-04-22T08:34:01.081396+0000","mapping_epoch":27,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22913024,"num_objects":5594,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14590","reported_seq":15044,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.830905+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:00.830905+0000","last_peered":"2024-04-22T08:34:00.830905+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:00.830905+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065810,"num_objects":5633,"num_object_clones":0,"num_object_copies":11266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14530,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14432","reported_seq":15313,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.138307+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:01.138307+0000","last_peered":"2024-04-22T08:34:01.138307+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:01.138307+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23569618,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14430,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2385}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14268","reported_seq":14798,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.904664+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:00.904664+0000","last_peered":"2024-04-22T08:34:00.904664+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:00.904664+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23187456,"num_objects":5661,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2102}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14363","reported_seq":22041,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.889762+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:00.889762+0000","last_peered":"2024-04-22T08:34:00.889762+0000","last_clean":"2024-04-22T08:34:00.889762+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:00.889762+0000","last_undegraded":"2024-04-22T08:34:00.889762+0000","last_fullsized":"2024-04-22T08:34:00.889762+0000","mapping_epoch":38,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22820050,"num_objects":5573,"num_object_clones":0,"num_object_copies":11146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14361,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14361","reported_seq":15116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.904077+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:00.904077+0000","last_peered":"2024-04-22T08:34:00.904077+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:00.904077+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23679593,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14360,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14252","reported_seq":14924,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.015867+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:01.015867+0000","last_peered":"2024-04-22T08:34:01.015867+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:01.015867+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22519808,"num_objects":5498,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14321","reported_seq":14923,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.146683+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:01.146683+0000","last_peered":"2024-04-22T08:34:01.146683+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:01.146683+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418066,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14319,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14446","reported_seq":15159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.117019+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:01.117019+0000","last_peered":"2024-04-22T08:34:01.117019+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:01.117019+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22759844,"num_objects":5560,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14443,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":369164141,"num_objects":90140,"num_object_clones":0,"num_object_copies":180280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203564,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12512,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925872128,"data_stored":925797883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117080,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618176,"kb_used_data":301880,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93753664,"statfs":{"total":96636764160,"available":96003751936,"internally_reserved":0,"allocated":309125120,"data_stored":308899193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629712,"kb_used_data":273224,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93742128,"statfs":{"total":96636764160,"available":95991939072,"internally_reserved":0,"allocated":279781376,"data_stored":279499785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36099999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313308,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715516,"kb_used_data":332840,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93656324,"statfs":{"total":96636764160,"available":95904075776,"internally_reserved":0,"allocated":340828160,"data_stored":340612767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,17,2,1,1,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.435},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339214336,"data_stored":339210857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278097920,"data_stored":278034705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308559872,"data_stored":308552321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:05.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:06.212 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:06.212 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:06.271 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-04-22T08:34:04.322308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":369164141,"num_objects":90140,"num_object_clones":0,"num_object_copies":180280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203564,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12512,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1963404,"kb_used_data":907944,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281152116,"statfs":{"total":289910292480,"available":287899766784,"internally_reserved":0,"allocated":929734656,"data_stored":929011745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1080740974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,17,3,9,1,6,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19025920,"num_objects":-4645,"num_object_clones":0,"num_object_copies":-9290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002820"},"pg_stats":[{"pgid":"2.f","version":"39'14455","reported_seq":15173,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.092080+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:01.092080+0000","last_peered":"2024-04-22T08:34:01.092080+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:01.092080+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23400448,"num_objects":5713,"num_object_clones":0,"num_object_copies":11426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14455,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2245}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14510","reported_seq":14628,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.898213+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:00.898213+0000","last_peered":"2024-04-22T08:34:00.898213+0000","last_clean":"2024-04-22T08:34:00.898213+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:00.898213+0000","last_undegraded":"2024-04-22T08:34:00.898213+0000","last_fullsized":"2024-04-22T08:34:00.898213+0000","mapping_epoch":27,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23437312,"num_objects":5722,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14284","reported_seq":14818,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.877653+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:00.877653+0000","last_peered":"2024-04-22T08:34:00.877653+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:00.877653+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851608,"num_objects":5580,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11111,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14493","reported_seq":14527,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.848512+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:00.848512+0000","last_peered":"2024-04-22T08:34:00.848512+0000","last_clean":"2024-04-22T08:34:00.848512+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:00.848512+0000","last_undegraded":"2024-04-22T08:34:00.848512+0000","last_fullsized":"2024-04-22T08:34:00.848512+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23023616,"num_objects":5621,"num_object_clones":0,"num_object_copies":11242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14493,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14340","reported_seq":14482,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:01.116113+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:01.116113+0000","last_peered":"2024-04-22T08:34:01.116113+0000","last_clean":"2024-04-22T08:34:01.116113+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:01.116113+0000","last_undegraded":"2024-04-22T08:34:01.116113+0000","last_fullsized":"2024-04-22T08:34:01.116113+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851584,"num_objects":5579,"num_object_clones":0,"num_object_copies":11158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14502","reported_seq":19653,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.142647+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:01.142647+0000","last_peered":"2024-04-22T08:34:01.142647+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:01.142647+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22712320,"num_objects":5545,"num_object_clones":0,"num_object_copies":11090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14260","reported_seq":21999,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.076814+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:00.076814+0000","last_peered":"2024-04-22T08:34:00.076814+0000","last_clean":"2024-04-22T08:34:00.076814+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:00.076814+0000","last_undegraded":"2024-04-22T08:34:00.076814+0000","last_fullsized":"2024-04-22T08:34:00.076814+0000","mapping_epoch":36,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":5604,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14212","reported_seq":14360,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:01.081396+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:01.081396+0000","last_peered":"2024-04-22T08:34:01.081396+0000","last_clean":"2024-04-22T08:34:01.081396+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:01.081396+0000","last_undegraded":"2024-04-22T08:34:01.081396+0000","last_fullsized":"2024-04-22T08:34:01.081396+0000","mapping_epoch":27,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22913024,"num_objects":5594,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14590","reported_seq":15044,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.830905+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:00.830905+0000","last_peered":"2024-04-22T08:34:00.830905+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:00.830905+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065810,"num_objects":5633,"num_object_clones":0,"num_object_copies":11266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14530,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14432","reported_seq":15313,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.138307+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:01.138307+0000","last_peered":"2024-04-22T08:34:01.138307+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:01.138307+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23569618,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14430,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2385}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14268","reported_seq":14798,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.904664+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:00.904664+0000","last_peered":"2024-04-22T08:34:00.904664+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:00.904664+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23187456,"num_objects":5661,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2102}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14363","reported_seq":22041,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:00.889762+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:00.889762+0000","last_peered":"2024-04-22T08:34:00.889762+0000","last_clean":"2024-04-22T08:34:00.889762+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:00.889762+0000","last_undegraded":"2024-04-22T08:34:00.889762+0000","last_fullsized":"2024-04-22T08:34:00.889762+0000","mapping_epoch":38,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22820050,"num_objects":5573,"num_object_clones":0,"num_object_copies":11146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14361,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14361","reported_seq":15116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:00.904077+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:00.904077+0000","last_peered":"2024-04-22T08:34:00.904077+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:00.904077+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23679593,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14360,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14252","reported_seq":14924,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.015867+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:01.015867+0000","last_peered":"2024-04-22T08:34:01.015867+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:01.015867+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4200","ondisk_log_start":"35'4200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22519808,"num_objects":5498,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14321","reported_seq":14923,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.146683+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:01.146683+0000","last_peered":"2024-04-22T08:34:01.146683+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:01.146683+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418066,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14319,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14446","reported_seq":15159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:01.117019+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:01.117019+0000","last_peered":"2024-04-22T08:34:01.117019+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:01.117019+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22759844,"num_objects":5560,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14443,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":369164141,"num_objects":90140,"num_object_clones":0,"num_object_copies":180280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203564,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12512,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925872128,"data_stored":925797883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117080,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618176,"kb_used_data":301880,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93753664,"statfs":{"total":96636764160,"available":96003751936,"internally_reserved":0,"allocated":309125120,"data_stored":308899193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.27700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629712,"kb_used_data":273224,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93742128,"statfs":{"total":96636764160,"available":95991939072,"internally_reserved":0,"allocated":279781376,"data_stored":279499785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.36099999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313308,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715516,"kb_used_data":332840,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93656324,"statfs":{"total":96636764160,"available":95904075776,"internally_reserved":0,"allocated":340828160,"data_stored":340612767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":391835347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,17,2,1,1,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.435},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339214336,"data_stored":339210857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278097920,"data_stored":278034705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308559872,"data_stored":308552321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:06.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14455", 'reported_seq': 15173, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.092080+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:01.092080+0000', 'last_peered': '2024-04-22T08:34:01.092080+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:01.092080+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4400", 'ondisk_log_start': "35'4400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23400448, 'num_objects': 5713, 'num_object_clones': 0, 'num_object_copies': 11426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14455, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2245}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'14284", 'reported_seq': 14818, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:00.877653+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:00.877653+0000', 'last_peered': '2024-04-22T08:34:00.877653+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:00.877653+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'4200", 'ondisk_log_start': "35'4200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22851608, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 11160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11111, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14502", 'reported_seq': 19653, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.142647+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:01.142647+0000', 'last_peered': '2024-04-22T08:34:01.142647+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:01.142647+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4500", 'ondisk_log_start': "35'4500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22712320, 'num_objects': 5545, 'num_object_clones': 0, 'num_object_copies': 11090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11371, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2493, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'14590", 'reported_seq': 15044, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:00.830905+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:00.830905+0000', 'last_peered': '2024-04-22T08:34:00.830905+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:00.830905+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'4500", 'ondisk_log_start': "35'4500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23065810, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 11266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14530, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:06.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14432", 'reported_seq': 15313, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.138307+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:01.138307+0000', 'last_peered': '2024-04-22T08:34:01.138307+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:01.138307+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'4400", 'ondisk_log_start': "35'4400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23569618, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 11512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14430, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2385}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'14268", 'reported_seq': 14798, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:00.904664+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:00.904664+0000', 'last_peered': '2024-04-22T08:34:00.904664+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:00.904664+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'4200", 'ondisk_log_start': "35'4200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23187456, 'num_objects': 5661, 'num_object_clones': 0, 'num_object_copies': 11322, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10722, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2102}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14361", 'reported_seq': 15116, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:00.904077+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:00.904077+0000', 'last_peered': '2024-04-22T08:34:00.904077+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:00.904077+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23679593, 'num_objects': 5782, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14360, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14252", 'reported_seq': 14924, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.015867+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:01.015867+0000', 'last_peered': '2024-04-22T08:34:01.015867+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:01.015867+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4200", 'ondisk_log_start': "35'4200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22519808, 'num_objects': 5498, 'num_object_clones': 0, 'num_object_copies': 10996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10483, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14321", 'reported_seq': 14923, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.146683+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:01.146683+0000', 'last_peered': '2024-04-22T08:34:01.146683+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:01.146683+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4300", 'ondisk_log_start': "35'4300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23418066, 'num_objects': 5719, 'num_object_clones': 0, 'num_object_copies': 11438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14319, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:06.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14446", 'reported_seq': 15159, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:01.117019+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:01.117019+0000', 'last_peered': '2024-04-22T08:34:01.117019+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:01.117019+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4400", 'ondisk_log_start': "35'4400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22759844, 'num_objects': 5560, 'num_object_clones': 0, 'num_object_copies': 11120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14443, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:06.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:07.075 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32993/175448 objects degraded (18.805%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":87724,"data_bytes":359268205,"bytes_used":2012532736,"bytes_avail":287897759744,"bytes_total":289910292480,"degraded_objects":32993,"degraded_total":175448,"degraded_ratio":0.18805002051890018,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":296,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":588},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:07.114 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:07.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:08.042 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:08.095 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-04-22T08:34:06.322736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359268205,"num_objects":87724,"num_object_clones":0,"num_object_copies":175448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205980,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12614,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160605,"ondisk_log_size":160605,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1965364,"kb_used_data":887764,"kb_used_omap":12,"kb_used_meta":1077555,"kb_avail":281150156,"statfs":{"total":289910292480,"available":287897759744,"internally_reserved":0,"allocated":909070336,"data_stored":908347425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1103416430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,11,7,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28921856,"num_objects":-7061,"num_object_clones":0,"num_object_copies":-14122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"2.f","version":"39'14602","reported_seq":15319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.146907+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:06.146907+0000","last_peered":"2024-04-22T08:34:06.146907+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:06.146907+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22798336,"num_objects":5566,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14602,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2098}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14664","reported_seq":14782,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.818376+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:05.818376+0000","last_peered":"2024-04-22T08:34:05.818376+0000","last_clean":"2024-04-22T08:34:05.818376+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:05.818376+0000","last_undegraded":"2024-04-22T08:34:05.818376+0000","last_fullsized":"2024-04-22T08:34:05.818376+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":5568,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14438","reported_seq":14972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.939627+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:05.939627+0000","last_peered":"2024-04-22T08:34:05.939627+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:05.939627+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22220824,"num_objects":5426,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11265,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14659","reported_seq":14693,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.892059+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:05.892059+0000","last_peered":"2024-04-22T08:34:05.892059+0000","last_clean":"2024-04-22T08:34:05.892059+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:05.892059+0000","last_undegraded":"2024-04-22T08:34:05.892059+0000","last_fullsized":"2024-04-22T08:34:05.892059+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22343680,"num_objects":5455,"num_object_clones":0,"num_object_copies":10910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14659,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14472","reported_seq":14614,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:06.126021+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:06.126021+0000","last_peered":"2024-04-22T08:34:06.126021+0000","last_clean":"2024-04-22T08:34:06.126021+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:06.126021+0000","last_undegraded":"2024-04-22T08:34:06.126021+0000","last_fullsized":"2024-04-22T08:34:06.126021+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22310912,"num_objects":5447,"num_object_clones":0,"num_object_copies":10894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14649","reported_seq":20004,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.180891+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:06.180891+0000","last_peered":"2024-04-22T08:34:06.180891+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:06.180891+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22110208,"num_objects":5398,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2595,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14402","reported_seq":22141,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.170953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:05.170953+0000","last_peered":"2024-04-22T08:34:05.170953+0000","last_clean":"2024-04-22T08:34:05.170953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:05.170953+0000","last_undegraded":"2024-04-22T08:34:05.170953+0000","last_fullsized":"2024-04-22T08:34:05.170953+0000","mapping_epoch":36,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22372352,"num_objects":5462,"num_object_clones":0,"num_object_copies":10924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14363","reported_seq":14511,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:06.114733+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:06.114733+0000","last_peered":"2024-04-22T08:34:06.114733+0000","last_clean":"2024-04-22T08:34:06.114733+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:06.114733+0000","last_undegraded":"2024-04-22T08:34:06.114733+0000","last_fullsized":"2024-04-22T08:34:06.114733+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22294528,"num_objects":5443,"num_object_clones":0,"num_object_copies":10886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14745","reported_seq":15202,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.852172+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:05.852172+0000","last_peered":"2024-04-22T08:34:05.852172+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:05.852172+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22430930,"num_objects":5478,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14685,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14573","reported_seq":15453,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.181344+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:06.181344+0000","last_peered":"2024-04-22T08:34:06.181344+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:06.181344+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22992082,"num_objects":5615,"num_object_clones":0,"num_object_copies":11230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14407","reported_seq":14939,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.803667+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:05.803667+0000","last_peered":"2024-04-22T08:34:05.803667+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:05.803667+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22618112,"num_objects":5522,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10861,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1963}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133556+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:18.133556+0000","last_peered":"2024-04-22T08:32:18.133556+0000","last_clean":"2024-04-22T08:32:18.133556+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:18.133556+0000","last_undegraded":"2024-04-22T08:32:18.133556+0000","last_fullsized":"2024-04-22T08:32:18.133556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14495","reported_seq":22173,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.908131+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:05.908131+0000","last_peered":"2024-04-22T08:34:05.908131+0000","last_clean":"2024-04-22T08:34:05.908131+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:05.908131+0000","last_undegraded":"2024-04-22T08:34:05.908131+0000","last_fullsized":"2024-04-22T08:34:05.908131+0000","mapping_epoch":38,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279378,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14493,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14525","reported_seq":15278,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.186714+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:06.186714+0000","last_peered":"2024-04-22T08:34:06.186714+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:06.186714+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23007849,"num_objects":5618,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14524,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133335+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:18.133335+0000","last_peered":"2024-04-22T08:32:18.133335+0000","last_clean":"2024-04-22T08:32:18.133335+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:18.133335+0000","last_undegraded":"2024-04-22T08:32:18.133335+0000","last_fullsized":"2024-04-22T08:32:18.133335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14407","reported_seq":15079,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.162845+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:06.162845+0000","last_peered":"2024-04-22T08:34:06.162845+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:06.162845+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21884928,"num_objects":5343,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14502","reported_seq":15104,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.178410+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:06.178410+0000","last_peered":"2024-04-22T08:34:06.178410+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:06.178410+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22676690,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14500,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133232+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:18.133232+0000","last_peered":"2024-04-22T08:32:18.133232+0000","last_clean":"2024-04-22T08:32:18.133232+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:18.133232+0000","last_undegraded":"2024-04-22T08:32:18.133232+0000","last_fullsized":"2024-04-22T08:32:18.133232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14602","reported_seq":15315,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.157434+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:06.157434+0000","last_peered":"2024-04-22T08:34:06.157434+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:06.157434+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22120868,"num_objects":5404,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":359268205,"num_objects":87724,"num_object_clones":0,"num_object_copies":175448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205980,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12614,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":906240000,"data_stored":906165755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160605,"ondisk_log_size":160605,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117081,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617452,"kb_used_data":297056,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93754388,"statfs":{"total":96636764160,"available":96004493312,"internally_reserved":0,"allocated":304185344,"data_stored":303959417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":3.9830000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.61299999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920843,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630900,"kb_used_data":266220,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93740940,"statfs":{"total":96636764160,"available":95990722560,"internally_reserved":0,"allocated":272609280,"data_stored":272327689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":373419724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.79600000000000004,"15min":0.59399999999999997},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.4780000000000002,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.221},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.88,"15min":0.94199999999999995},"min":{"1min":0.255,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.8860000000000001,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.254}]},{"osd":2,"last update":"Mon Apr 22 08:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.625,"15min":1.4570000000000001},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.9560000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.377},{"interface":"front","average":{"1min":1.333,"5min":1.492,"15min":1.264},"min":{"1min":0.33500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.9299999999999997,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.40799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313309,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717012,"kb_used_data":324488,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93654828,"statfs":{"total":96636764160,"available":95902543872,"internally_reserved":0,"allocated":332275712,"data_stored":332060319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331759616,"data_stored":331756137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270753792,"data_stored":270690577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303726592,"data_stored":303719041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:11.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:11.684 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:11.686 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:11.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-04-22T08:34:10.323674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358653805,"num_objects":87574,"num_object_clones":0,"num_object_copies":175148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87574,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206130,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12614,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160655,"ondisk_log_size":160655,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1948660,"kb_used_data":879252,"kb_used_omap":12,"kb_used_meta":1069363,"kb_avail":281166860,"statfs":{"total":289910292480,"available":287914864640,"internally_reserved":0,"allocated":900354048,"data_stored":899631137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1095027822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,8,10,19,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20074496,"num_objects":-4901,"num_object_clones":0,"num_object_copies":-9802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002768"},"pg_stats":[{"pgid":"2.f","version":"39'14602","reported_seq":15319,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.146907+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:06.146907+0000","last_peered":"2024-04-22T08:34:06.146907+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:06.146907+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22798336,"num_objects":5566,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14602,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2098}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14664","reported_seq":14782,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.818376+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:05.818376+0000","last_peered":"2024-04-22T08:34:05.818376+0000","last_clean":"2024-04-22T08:34:05.818376+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:05.818376+0000","last_undegraded":"2024-04-22T08:34:05.818376+0000","last_fullsized":"2024-04-22T08:34:05.818376+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":5568,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14438","reported_seq":14972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.939627+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:05.939627+0000","last_peered":"2024-04-22T08:34:05.939627+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:05.939627+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22220824,"num_objects":5426,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11265,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14659","reported_seq":14693,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.892059+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:05.892059+0000","last_peered":"2024-04-22T08:34:05.892059+0000","last_clean":"2024-04-22T08:34:05.892059+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:05.892059+0000","last_undegraded":"2024-04-22T08:34:05.892059+0000","last_fullsized":"2024-04-22T08:34:05.892059+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22343680,"num_objects":5455,"num_object_clones":0,"num_object_copies":10910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14659,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14472","reported_seq":14614,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:06.126021+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:06.126021+0000","last_peered":"2024-04-22T08:34:06.126021+0000","last_clean":"2024-04-22T08:34:06.126021+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:06.126021+0000","last_undegraded":"2024-04-22T08:34:06.126021+0000","last_fullsized":"2024-04-22T08:34:06.126021+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22310912,"num_objects":5447,"num_object_clones":0,"num_object_copies":10894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14649","reported_seq":20004,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.180891+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:06.180891+0000","last_peered":"2024-04-22T08:34:06.180891+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:06.180891+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22110208,"num_objects":5398,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2595,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14552","reported_seq":22291,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.228525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:10.228525+0000","last_peered":"2024-04-22T08:34:10.228525+0000","last_clean":"2024-04-22T08:34:10.228525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:10.228525+0000","last_undegraded":"2024-04-22T08:34:10.228525+0000","last_fullsized":"2024-04-22T08:34:10.228525+0000","mapping_epoch":36,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":5312,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14363","reported_seq":14511,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:06.114733+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:06.114733+0000","last_peered":"2024-04-22T08:34:06.114733+0000","last_clean":"2024-04-22T08:34:06.114733+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:06.114733+0000","last_undegraded":"2024-04-22T08:34:06.114733+0000","last_fullsized":"2024-04-22T08:34:06.114733+0000","mapping_epoch":27,"log_start":"35'4300","ondisk_log_start":"35'4300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22294528,"num_objects":5443,"num_object_clones":0,"num_object_copies":10886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14745","reported_seq":15202,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.852172+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:05.852172+0000","last_peered":"2024-04-22T08:34:05.852172+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:05.852172+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22430930,"num_objects":5478,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14685,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14573","reported_seq":15453,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.181344+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:06.181344+0000","last_peered":"2024-04-22T08:34:06.181344+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:06.181344+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22992082,"num_objects":5615,"num_object_clones":0,"num_object_copies":11230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14407","reported_seq":14939,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:05.803667+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:05.803667+0000","last_peered":"2024-04-22T08:34:05.803667+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:05.803667+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22618112,"num_objects":5522,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10861,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1963}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14495","reported_seq":22173,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:05.908131+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:05.908131+0000","last_peered":"2024-04-22T08:34:05.908131+0000","last_clean":"2024-04-22T08:34:05.908131+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:05.908131+0000","last_undegraded":"2024-04-22T08:34:05.908131+0000","last_fullsized":"2024-04-22T08:34:05.908131+0000","mapping_epoch":38,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279378,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14493,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14525","reported_seq":15278,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.186714+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:06.186714+0000","last_peered":"2024-04-22T08:34:06.186714+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:06.186714+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23007849,"num_objects":5618,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14524,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14407","reported_seq":15079,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.162845+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:06.162845+0000","last_peered":"2024-04-22T08:34:06.162845+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:06.162845+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21884928,"num_objects":5343,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14502","reported_seq":15104,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.178410+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:06.178410+0000","last_peered":"2024-04-22T08:34:06.178410+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:06.178410+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22676690,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14500,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14602","reported_seq":15315,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:06.157434+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:06.157434+0000","last_peered":"2024-04-22T08:34:06.157434+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:06.157434+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22120868,"num_objects":5404,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":358653805,"num_objects":87574,"num_object_clones":0,"num_object_copies":175148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87574,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206130,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12614,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898908160,"data_stored":898833915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160655,"ondisk_log_size":160655,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117081,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617452,"kb_used_data":297056,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93754388,"statfs":{"total":96636764160,"available":96004493312,"internally_reserved":0,"allocated":304185344,"data_stored":303959417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":3.9830000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.3720000000000001,"15min":1.081},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":6.1059999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.4299999999999999,"15min":0.85999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":18.32,"5min":18.32,"15min":18.32},"last":0.61299999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614196,"kb_used_data":257708,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93757644,"statfs":{"total":96636764160,"available":96007827456,"internally_reserved":0,"allocated":263892992,"data_stored":263611401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.8620000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.21299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313309,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717012,"kb_used_data":324488,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93654828,"statfs":{"total":96636764160,"available":95902543872,"internally_reserved":0,"allocated":332275712,"data_stored":332060319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331759616,"data_stored":331756137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263421952,"data_stored":263358737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303726592,"data_stored":303719041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:11.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:12.517 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-04-22T08:34:12.324116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":349671277,"num_objects":85381,"num_object_clones":0,"num_object_copies":170762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208323,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12727,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1949652,"kb_used_data":867964,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281165868,"statfs":{"total":289910292480,"available":287913848832,"internally_reserved":0,"allocated":888795136,"data_stored":888072225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,7,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28409856,"num_objects":-6936,"num_object_clones":0,"num_object_copies":-13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6936,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002780"},"pg_stats":[{"pgid":"2.f","version":"39'14763","reported_seq":15482,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.257790+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:11.257790+0000","last_peered":"2024-04-22T08:34:11.257790+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:11.257790+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138880,"num_objects":5405,"num_object_clones":0,"num_object_copies":10810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14763,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1937}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14790","reported_seq":14908,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.915633+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:10.915633+0000","last_peered":"2024-04-22T08:34:10.915633+0000","last_clean":"2024-04-22T08:34:10.915633+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:10.915633+0000","last_undegraded":"2024-04-22T08:34:10.915633+0000","last_fullsized":"2024-04-22T08:34:10.915633+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":5442,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14584","reported_seq":15118,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.923697+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:10.923697+0000","last_peered":"2024-04-22T08:34:10.923697+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:10.923697+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21622808,"num_objects":5280,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11411,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14810","reported_seq":14842,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.902949+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:10.902949+0000","last_peered":"2024-04-22T08:34:10.902949+0000","last_clean":"2024-04-22T08:34:10.902949+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:10.902949+0000","last_undegraded":"2024-04-22T08:34:10.902949+0000","last_fullsized":"2024-04-22T08:34:10.902949+0000","mapping_epoch":27,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21725184,"num_objects":5304,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14810,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14619","reported_seq":14761,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:11.239085+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:11.239085+0000","last_peered":"2024-04-22T08:34:11.239085+0000","last_clean":"2024-04-22T08:34:11.239085+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:11.239085+0000","last_undegraded":"2024-04-22T08:34:11.239085+0000","last_fullsized":"2024-04-22T08:34:11.239085+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21708800,"num_objects":5300,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14770","reported_seq":20351,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.240160+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:11.240160+0000","last_peered":"2024-04-22T08:34:11.240160+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:11.240160+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21614592,"num_objects":5277,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11639,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2708,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14552","reported_seq":22291,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.228525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:10.228525+0000","last_peered":"2024-04-22T08:34:10.228525+0000","last_clean":"2024-04-22T08:34:10.228525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:10.228525+0000","last_undegraded":"2024-04-22T08:34:10.228525+0000","last_fullsized":"2024-04-22T08:34:10.228525+0000","mapping_epoch":36,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":5312,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14485","reported_seq":14633,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:11.160944+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:11.160944+0000","last_peered":"2024-04-22T08:34:11.160944+0000","last_clean":"2024-04-22T08:34:11.160944+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:11.160944+0000","last_undegraded":"2024-04-22T08:34:11.160944+0000","last_fullsized":"2024-04-22T08:34:11.160944+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21794816,"num_objects":5321,"num_object_clones":0,"num_object_copies":10642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14908","reported_seq":15365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.973392+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:10.973392+0000","last_peered":"2024-04-22T08:34:10.973392+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:10.973392+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763282,"num_objects":5315,"num_object_clones":0,"num_object_copies":10630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14848,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14735","reported_seq":15616,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.218544+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:11.218544+0000","last_peered":"2024-04-22T08:34:11.218544+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:11.218544+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328530,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14733,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2082}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14564","reported_seq":15096,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.900775+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:10.900775+0000","last_peered":"2024-04-22T08:34:10.900775+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:10.900775+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14646","reported_seq":22324,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.946142+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:10.946142+0000","last_peered":"2024-04-22T08:34:10.946142+0000","last_clean":"2024-04-22T08:34:10.946142+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:10.946142+0000","last_undegraded":"2024-04-22T08:34:10.946142+0000","last_fullsized":"2024-04-22T08:34:10.946142+0000","mapping_epoch":38,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21660882,"num_objects":5290,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14644,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14659","reported_seq":15414,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.131491+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:11.131491+0000","last_peered":"2024-04-22T08:34:11.131491+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:11.131491+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22458985,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14658,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14571","reported_seq":15243,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.185992+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:11.185992+0000","last_peered":"2024-04-22T08:34:11.185992+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:11.185992+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21213184,"num_objects":5179,"num_object_clones":0,"num_object_copies":10358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14638","reported_seq":15240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.181989+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:11.181989+0000","last_peered":"2024-04-22T08:34:11.181989+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:11.181989+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22119634,"num_objects":5402,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14636,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14754","reported_seq":15468,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.251641+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:11.251641+0000","last_peered":"2024-04-22T08:34:11.251641+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:11.251641+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21498276,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14751,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":349671277,"num_objects":85381,"num_object_clones":0,"num_object_copies":170762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208323,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12727,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887091200,"data_stored":887016955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117082,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617048,"kb_used_data":292560,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93754792,"statfs":{"total":96636764160,"available":96004907008,"internally_reserved":0,"allocated":299581440,"data_stored":299355513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.254},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.53500000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614196,"kb_used_data":257708,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93757644,"statfs":{"total":96636764160,"available":96007827456,"internally_reserved":0,"allocated":263892992,"data_stored":263611401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.8620000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.21299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313310,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718408,"kb_used_data":317696,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93653432,"statfs":{"total":96636764160,"available":95901114368,"internally_reserved":0,"allocated":325320704,"data_stored":325105311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.496}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324648960,"data_stored":324645481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263421952,"data_stored":263358737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299020288,"data_stored":299012737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:12.518 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14763", 'reported_seq': 15482, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.257790+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:11.257790+0000', 'last_peered': '2024-04-22T08:34:11.257790+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:11.257790+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22138880, 'num_objects': 5405, 'num_object_clones': 0, 'num_object_copies': 10810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14763, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1937}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'14584", 'reported_seq': 15118, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:10.923697+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:10.923697+0000', 'last_peered': '2024-04-22T08:34:10.923697+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:10.923697+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'4500", 'ondisk_log_start': "35'4500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21622808, 'num_objects': 5280, 'num_object_clones': 0, 'num_object_copies': 10560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5280, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11411, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14770", 'reported_seq': 20351, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.240160+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:11.240160+0000', 'last_peered': '2024-04-22T08:34:11.240160+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:11.240160+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21614592, 'num_objects': 5277, 'num_object_clones': 0, 'num_object_copies': 10554, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1174, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11639, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2708, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'14908", 'reported_seq': 15365, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:10.973392+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:10.973392+0000', 'last_peered': '2024-04-22T08:34:10.973392+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:10.973392+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21763282, 'num_objects': 5315, 'num_object_clones': 0, 'num_object_copies': 10630, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14848, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14735", 'reported_seq': 15616, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.218544+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:11.218544+0000', 'last_peered': '2024-04-22T08:34:11.218544+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:11.218544+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22328530, 'num_objects': 5453, 'num_object_clones': 0, 'num_object_copies': 10906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14733, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2082}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:12.519 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'14564", 'reported_seq': 15096, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:10.900775+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:10.900775+0000', 'last_peered': '2024-04-22T08:34:10.900775+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:10.900775+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'4500", 'ondisk_log_start': "35'4500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21975040, 'num_objects': 5365, 'num_object_clones': 0, 'num_object_copies': 10730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11018, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1806}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14659", 'reported_seq': 15414, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.131491+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:11.131491+0000', 'last_peered': '2024-04-22T08:34:11.131491+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:11.131491+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4600", 'ondisk_log_start': "35'4600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22458985, 'num_objects': 5484, 'num_object_clones': 0, 'num_object_copies': 10968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14658, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14571", 'reported_seq': 15243, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.185992+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:11.185992+0000', 'last_peered': '2024-04-22T08:34:11.185992+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:11.185992+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4500", 'ondisk_log_start': "35'4500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21213184, 'num_objects': 5179, 'num_object_clones': 0, 'num_object_copies': 10358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10802, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14638", 'reported_seq': 15240, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.181989+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:11.181989+0000', 'last_peered': '2024-04-22T08:34:11.181989+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:11.181989+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4600", 'ondisk_log_start': "35'4600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22119634, 'num_objects': 5402, 'num_object_clones': 0, 'num_object_copies': 10804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14636, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:12.520 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14754", 'reported_seq': 15468, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:11.251641+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:11.251641+0000', 'last_peered': '2024-04-22T08:34:11.251641+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:11.251641+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21498276, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 10504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14751, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:12.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:13.549 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:13.549 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32880/170762 objects degraded (19.255%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":488,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":85381,"data_bytes":349671277,"bytes_used":1996443648,"bytes_avail":287913848832,"bytes_total":289910292480,"degraded_objects":32880,"degraded_total":170762,"degraded_ratio":0.19254869350323842,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":310,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":577},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:13.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:13.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:14.489 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:14.491 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:14.545 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-04-22T08:34:14.324586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":349671277,"num_objects":85381,"num_object_clones":0,"num_object_copies":170762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208323,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12727,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1949652,"kb_used_data":867964,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281165868,"statfs":{"total":289910292480,"available":287913848832,"internally_reserved":0,"allocated":888795136,"data_stored":888072225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,7,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19492864,"num_objects":-4759,"num_object_clones":0,"num_object_copies":-9518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002786"},"pg_stats":[{"pgid":"2.f","version":"39'14763","reported_seq":15482,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.257790+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:11.257790+0000","last_peered":"2024-04-22T08:34:11.257790+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:11.257790+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138880,"num_objects":5405,"num_object_clones":0,"num_object_copies":10810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14763,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1937}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14790","reported_seq":14908,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.915633+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:10.915633+0000","last_peered":"2024-04-22T08:34:10.915633+0000","last_clean":"2024-04-22T08:34:10.915633+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:10.915633+0000","last_undegraded":"2024-04-22T08:34:10.915633+0000","last_fullsized":"2024-04-22T08:34:10.915633+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":5442,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14584","reported_seq":15118,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.923697+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:10.923697+0000","last_peered":"2024-04-22T08:34:10.923697+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:10.923697+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21622808,"num_objects":5280,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11411,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14810","reported_seq":14842,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.902949+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:10.902949+0000","last_peered":"2024-04-22T08:34:10.902949+0000","last_clean":"2024-04-22T08:34:10.902949+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:10.902949+0000","last_undegraded":"2024-04-22T08:34:10.902949+0000","last_fullsized":"2024-04-22T08:34:10.902949+0000","mapping_epoch":27,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21725184,"num_objects":5304,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14810,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14619","reported_seq":14761,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:11.239085+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:11.239085+0000","last_peered":"2024-04-22T08:34:11.239085+0000","last_clean":"2024-04-22T08:34:11.239085+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:11.239085+0000","last_undegraded":"2024-04-22T08:34:11.239085+0000","last_fullsized":"2024-04-22T08:34:11.239085+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21708800,"num_objects":5300,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14770","reported_seq":20351,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.240160+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:11.240160+0000","last_peered":"2024-04-22T08:34:11.240160+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:11.240160+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21614592,"num_objects":5277,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11639,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2708,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14552","reported_seq":22291,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.228525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:10.228525+0000","last_peered":"2024-04-22T08:34:10.228525+0000","last_clean":"2024-04-22T08:34:10.228525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:10.228525+0000","last_undegraded":"2024-04-22T08:34:10.228525+0000","last_fullsized":"2024-04-22T08:34:10.228525+0000","mapping_epoch":36,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":5312,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14485","reported_seq":14633,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:11.160944+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:11.160944+0000","last_peered":"2024-04-22T08:34:11.160944+0000","last_clean":"2024-04-22T08:34:11.160944+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:11.160944+0000","last_undegraded":"2024-04-22T08:34:11.160944+0000","last_fullsized":"2024-04-22T08:34:11.160944+0000","mapping_epoch":27,"log_start":"35'4400","ondisk_log_start":"35'4400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21794816,"num_objects":5321,"num_object_clones":0,"num_object_copies":10642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'14908","reported_seq":15365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.973392+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:10.973392+0000","last_peered":"2024-04-22T08:34:10.973392+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:10.973392+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763282,"num_objects":5315,"num_object_clones":0,"num_object_copies":10630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14848,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14735","reported_seq":15616,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.218544+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:11.218544+0000","last_peered":"2024-04-22T08:34:11.218544+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:11.218544+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328530,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14733,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2082}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14564","reported_seq":15096,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:10.900775+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:10.900775+0000","last_peered":"2024-04-22T08:34:10.900775+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:10.900775+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14646","reported_seq":22324,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:10.946142+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:10.946142+0000","last_peered":"2024-04-22T08:34:10.946142+0000","last_clean":"2024-04-22T08:34:10.946142+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:10.946142+0000","last_undegraded":"2024-04-22T08:34:10.946142+0000","last_fullsized":"2024-04-22T08:34:10.946142+0000","mapping_epoch":38,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21660882,"num_objects":5290,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14644,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14659","reported_seq":15414,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.131491+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:11.131491+0000","last_peered":"2024-04-22T08:34:11.131491+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:11.131491+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22458985,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14658,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14571","reported_seq":15243,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.185992+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:11.185992+0000","last_peered":"2024-04-22T08:34:11.185992+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:11.185992+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4500","ondisk_log_start":"35'4500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21213184,"num_objects":5179,"num_object_clones":0,"num_object_copies":10358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14638","reported_seq":15240,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.181989+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:11.181989+0000","last_peered":"2024-04-22T08:34:11.181989+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:11.181989+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22119634,"num_objects":5402,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14636,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14754","reported_seq":15468,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:11.251641+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:11.251641+0000","last_peered":"2024-04-22T08:34:11.251641+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:11.251641+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21498276,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14751,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":349671277,"num_objects":85381,"num_object_clones":0,"num_object_copies":170762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208323,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12727,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887091200,"data_stored":887016955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117082,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617048,"kb_used_data":292560,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93754792,"statfs":{"total":96636764160,"available":96004907008,"internally_reserved":0,"allocated":299581440,"data_stored":299355513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.254},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.53500000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614196,"kb_used_data":257708,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93757644,"statfs":{"total":96636764160,"available":96007827456,"internally_reserved":0,"allocated":263892992,"data_stored":263611401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":3.8620000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.21299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313310,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718408,"kb_used_data":317696,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93653432,"statfs":{"total":96636764160,"available":95901114368,"internally_reserved":0,"allocated":325320704,"data_stored":325105311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.496}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324648960,"data_stored":324645481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263421952,"data_stored":263358737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299020288,"data_stored":299012737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:17.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:18.408 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:18.411 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:18.456 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-04-22T08:34:18.325493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1931188,"kb_used_data":849492,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281184332,"statfs":{"total":289910292480,"available":287932755968,"internally_reserved":0,"allocated":869879808,"data_stored":869156897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,6,8,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19116032,"num_objects":-4667,"num_object_clones":0,"num_object_copies":-9334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"2.f","version":"39'14915","reported_seq":15632,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255653+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:16.255653+0000","last_peered":"2024-04-22T08:34:16.255653+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:16.255653+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21516288,"num_objects":5253,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14947","reported_seq":15065,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.967714+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:15.967714+0000","last_peered":"2024-04-22T08:34:15.967714+0000","last_clean":"2024-04-22T08:34:15.967714+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:15.967714+0000","last_undegraded":"2024-04-22T08:34:15.967714+0000","last_fullsized":"2024-04-22T08:34:15.967714+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647360,"num_objects":5285,"num_object_clones":0,"num_object_copies":10570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14711","reported_seq":15245,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.902478+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:15.902478+0000","last_peered":"2024-04-22T08:34:15.902478+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:15.902478+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102616,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11538,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14915","reported_seq":14948,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.949408+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:15.949408+0000","last_peered":"2024-04-22T08:34:15.949408+0000","last_clean":"2024-04-22T08:34:15.949408+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:15.949408+0000","last_undegraded":"2024-04-22T08:34:15.949408+0000","last_fullsized":"2024-04-22T08:34:15.949408+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21295104,"num_objects":5199,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14776","reported_seq":14918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.236745+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:16.236745+0000","last_peered":"2024-04-22T08:34:16.236745+0000","last_clean":"2024-04-22T08:34:16.236745+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:16.236745+0000","last_undegraded":"2024-04-22T08:34:16.236745+0000","last_fullsized":"2024-04-22T08:34:16.236745+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21065728,"num_objects":5143,"num_object_clones":0,"num_object_copies":10286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14912","reported_seq":20705,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.147505+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:16.147505+0000","last_peered":"2024-04-22T08:34:16.147505+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:16.147505+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21032960,"num_objects":5135,"num_object_clones":0,"num_object_copies":10270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14699","reported_seq":22438,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.330555+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:15.330555+0000","last_peered":"2024-04-22T08:34:15.330555+0000","last_clean":"2024-04-22T08:34:15.330555+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:15.330555+0000","last_undegraded":"2024-04-22T08:34:15.330555+0000","last_fullsized":"2024-04-22T08:34:15.330555+0000","mapping_epoch":36,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21155840,"num_objects":5165,"num_object_clones":0,"num_object_copies":10330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14645","reported_seq":14792,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.280397+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:16.280397+0000","last_peered":"2024-04-22T08:34:16.280397+0000","last_clean":"2024-04-22T08:34:16.280397+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:16.280397+0000","last_undegraded":"2024-04-22T08:34:16.280397+0000","last_fullsized":"2024-04-22T08:34:16.280397+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21139456,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15048","reported_seq":15505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.889802+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:15.889802+0000","last_peered":"2024-04-22T08:34:15.889802+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:15.889802+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189842,"num_objects":5175,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14988,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14878","reported_seq":15756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.298131+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:16.298131+0000","last_peered":"2024-04-22T08:34:16.298131+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:16.298131+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21742802,"num_objects":5310,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14876,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14728","reported_seq":15260,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.914578+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:15.914578+0000","last_peered":"2024-04-22T08:34:15.914578+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:15.914578+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21303296,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1642}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14802","reported_seq":22480,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.921662+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:15.921662+0000","last_peered":"2024-04-22T08:34:15.921662+0000","last_clean":"2024-04-22T08:34:15.921662+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:15.921662+0000","last_undegraded":"2024-04-22T08:34:15.921662+0000","last_fullsized":"2024-04-22T08:34:15.921662+0000","mapping_epoch":38,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21021906,"num_objects":5134,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14806","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.218914+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:16.218914+0000","last_peered":"2024-04-22T08:34:16.218914+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:16.218914+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21856873,"num_objects":5337,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14805,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14709","reported_seq":15381,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.256479+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:16.256479+0000","last_peered":"2024-04-22T08:34:16.256479+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:16.256479+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14794","reported_seq":15396,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.264681+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:16.264681+0000","last_peered":"2024-04-22T08:34:16.264681+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:16.264681+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21480658,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14792,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14887","reported_seq":15601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255748+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:16.255748+0000","last_peered":"2024-04-22T08:34:16.255748+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:16.255748+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20953508,"num_objects":5119,"num_object_clones":0,"num_object_copies":10238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867975168,"data_stored":867900923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117083,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616484,"kb_used_data":287900,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93755356,"statfs":{"total":96636764160,"available":96005484544,"internally_reserved":0,"allocated":294809600,"data_stored":294583673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.248}]}]},{"osd":1,"up_from":33,"seq":141733920845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615424,"kb_used_data":250736,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93756416,"statfs":{"total":96636764160,"available":96006569984,"internally_reserved":0,"allocated":256753664,"data_stored":256472073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":373419724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.372}]}]},{"osd":0,"up_from":21,"seq":90194313311,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699280,"kb_used_data":310856,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93672560,"statfs":{"total":96636764160,"available":95920701440,"internally_reserved":0,"allocated":318316544,"data_stored":318101151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":11.368},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.216},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317489152,"data_stored":317485673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256225280,"data_stored":256162065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294260736,"data_stored":294253185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:18.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:19.202 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:19.263 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-04-22T08:34:18.325493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1931188,"kb_used_data":849492,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281184332,"statfs":{"total":289910292480,"available":287932755968,"internally_reserved":0,"allocated":869879808,"data_stored":869156897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,6,8,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19116032,"num_objects":-4667,"num_object_clones":0,"num_object_copies":-9334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"2.f","version":"39'14915","reported_seq":15632,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255653+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:16.255653+0000","last_peered":"2024-04-22T08:34:16.255653+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:16.255653+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21516288,"num_objects":5253,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14947","reported_seq":15065,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.967714+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:15.967714+0000","last_peered":"2024-04-22T08:34:15.967714+0000","last_clean":"2024-04-22T08:34:15.967714+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:15.967714+0000","last_undegraded":"2024-04-22T08:34:15.967714+0000","last_fullsized":"2024-04-22T08:34:15.967714+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647360,"num_objects":5285,"num_object_clones":0,"num_object_copies":10570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14711","reported_seq":15245,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.902478+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:15.902478+0000","last_peered":"2024-04-22T08:34:15.902478+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:15.902478+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102616,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11538,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14915","reported_seq":14948,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.949408+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:15.949408+0000","last_peered":"2024-04-22T08:34:15.949408+0000","last_clean":"2024-04-22T08:34:15.949408+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:15.949408+0000","last_undegraded":"2024-04-22T08:34:15.949408+0000","last_fullsized":"2024-04-22T08:34:15.949408+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21295104,"num_objects":5199,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14776","reported_seq":14918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.236745+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:16.236745+0000","last_peered":"2024-04-22T08:34:16.236745+0000","last_clean":"2024-04-22T08:34:16.236745+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:16.236745+0000","last_undegraded":"2024-04-22T08:34:16.236745+0000","last_fullsized":"2024-04-22T08:34:16.236745+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21065728,"num_objects":5143,"num_object_clones":0,"num_object_copies":10286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14912","reported_seq":20705,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.147505+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:16.147505+0000","last_peered":"2024-04-22T08:34:16.147505+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:16.147505+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21032960,"num_objects":5135,"num_object_clones":0,"num_object_copies":10270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14699","reported_seq":22438,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.330555+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:15.330555+0000","last_peered":"2024-04-22T08:34:15.330555+0000","last_clean":"2024-04-22T08:34:15.330555+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:15.330555+0000","last_undegraded":"2024-04-22T08:34:15.330555+0000","last_fullsized":"2024-04-22T08:34:15.330555+0000","mapping_epoch":36,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21155840,"num_objects":5165,"num_object_clones":0,"num_object_copies":10330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14645","reported_seq":14792,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.280397+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:16.280397+0000","last_peered":"2024-04-22T08:34:16.280397+0000","last_clean":"2024-04-22T08:34:16.280397+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:16.280397+0000","last_undegraded":"2024-04-22T08:34:16.280397+0000","last_fullsized":"2024-04-22T08:34:16.280397+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21139456,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15048","reported_seq":15505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.889802+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:15.889802+0000","last_peered":"2024-04-22T08:34:15.889802+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:15.889802+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189842,"num_objects":5175,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14988,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14878","reported_seq":15756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.298131+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:16.298131+0000","last_peered":"2024-04-22T08:34:16.298131+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:16.298131+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21742802,"num_objects":5310,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14876,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14728","reported_seq":15260,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.914578+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:15.914578+0000","last_peered":"2024-04-22T08:34:15.914578+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:15.914578+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21303296,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1642}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14802","reported_seq":22480,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.921662+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:15.921662+0000","last_peered":"2024-04-22T08:34:15.921662+0000","last_clean":"2024-04-22T08:34:15.921662+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:15.921662+0000","last_undegraded":"2024-04-22T08:34:15.921662+0000","last_fullsized":"2024-04-22T08:34:15.921662+0000","mapping_epoch":38,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21021906,"num_objects":5134,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14806","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.218914+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:16.218914+0000","last_peered":"2024-04-22T08:34:16.218914+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:16.218914+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21856873,"num_objects":5337,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14805,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14709","reported_seq":15381,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.256479+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:16.256479+0000","last_peered":"2024-04-22T08:34:16.256479+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:16.256479+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14794","reported_seq":15396,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.264681+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:16.264681+0000","last_peered":"2024-04-22T08:34:16.264681+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:16.264681+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21480658,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14792,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14887","reported_seq":15601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255748+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:16.255748+0000","last_peered":"2024-04-22T08:34:16.255748+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:16.255748+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20953508,"num_objects":5119,"num_object_clones":0,"num_object_copies":10238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867975168,"data_stored":867900923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117083,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616484,"kb_used_data":287900,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93755356,"statfs":{"total":96636764160,"available":96005484544,"internally_reserved":0,"allocated":294809600,"data_stored":294583673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.248}]}]},{"osd":1,"up_from":33,"seq":141733920845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615424,"kb_used_data":250736,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93756416,"statfs":{"total":96636764160,"available":96006569984,"internally_reserved":0,"allocated":256753664,"data_stored":256472073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":373419724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.372}]}]},{"osd":0,"up_from":21,"seq":90194313311,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699280,"kb_used_data":310856,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93672560,"statfs":{"total":96636764160,"available":95920701440,"internally_reserved":0,"allocated":318316544,"data_stored":318101151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":11.368},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.216},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317489152,"data_stored":317485673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256225280,"data_stored":256162065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294260736,"data_stored":294253185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:19.264 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'14915", 'reported_seq': 15632, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.255653+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:16.255653+0000', 'last_peered': '2024-04-22T08:34:16.255653+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:16.255653+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21516288, 'num_objects': 5253, 'num_object_clones': 0, 'num_object_copies': 10506, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14915, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1785}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'14711", 'reported_seq': 15245, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:15.902478+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:15.902478+0000', 'last_peered': '2024-04-22T08:34:15.902478+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:15.902478+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21102616, 'num_objects': 5153, 'num_object_clones': 0, 'num_object_copies': 10306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5153, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11538, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'14912", 'reported_seq': 20705, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.147505+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:16.147505+0000', 'last_peered': '2024-04-22T08:34:16.147505+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:16.147505+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21032960, 'num_objects': 5135, 'num_object_clones': 0, 'num_object_copies': 10270, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1068, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11781, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2814, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4067}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'15048", 'reported_seq': 15505, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:15.889802+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:15.889802+0000', 'last_peered': '2024-04-22T08:34:15.889802+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:15.889802+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21189842, 'num_objects': 5175, 'num_object_clones': 0, 'num_object_copies': 10350, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14988, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:19.265 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'14878", 'reported_seq': 15756, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.298131+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:16.298131+0000', 'last_peered': '2024-04-22T08:34:16.298131+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:16.298131+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21742802, 'num_objects': 5310, 'num_object_clones': 0, 'num_object_copies': 10620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14876, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1939}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'14728", 'reported_seq': 15260, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:15.914578+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:15.914578+0000', 'last_peered': '2024-04-22T08:34:15.914578+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:15.914578+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21303296, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 10402, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11182, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1642}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14806", 'reported_seq': 15561, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.218914+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:16.218914+0000', 'last_peered': '2024-04-22T08:34:16.218914+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:16.218914+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21856873, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 10674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14805, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14709", 'reported_seq': 15381, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.256479+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:16.256479+0000', 'last_peered': '2024-04-22T08:34:16.256479+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:16.256479+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20647936, 'num_objects': 5041, 'num_object_clones': 0, 'num_object_copies': 10082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10940, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14794", 'reported_seq': 15396, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.264681+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:16.264681+0000', 'last_peered': '2024-04-22T08:34:16.264681+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:16.264681+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4700", 'ondisk_log_start': "35'4700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21480658, 'num_objects': 5246, 'num_object_clones': 0, 'num_object_copies': 10492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14792, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:19.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'14887", 'reported_seq': 15601, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:16.255748+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:16.255748+0000', 'last_peered': '2024-04-22T08:34:16.255748+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:16.255748+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20953508, 'num_objects': 5119, 'num_object_clones': 0, 'num_object_copies': 10238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14884, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:19.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:20.197 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:20.197 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32774/166114 objects degraded (19.730%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":495,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":83057,"data_bytes":340152173,"bytes_used":1977536512,"bytes_avail":287932755968,"bytes_total":289910292480,"degraded_objects":32774,"degraded_total":166114,"degraded_ratio":0.19729824096704676,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":388},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:20.239 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:20.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:21.059 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:21.060 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:21.109 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-04-22T08:34:20.325981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1931188,"kb_used_data":849492,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281184332,"statfs":{"total":289910292480,"available":287932755968,"internally_reserved":0,"allocated":869879808,"data_stored":869156897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,6,8,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19116032,"num_objects":-4667,"num_object_clones":0,"num_object_copies":-9334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"2.f","version":"39'14915","reported_seq":15632,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255653+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:16.255653+0000","last_peered":"2024-04-22T08:34:16.255653+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:16.255653+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21516288,"num_objects":5253,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'14947","reported_seq":15065,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.967714+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:15.967714+0000","last_peered":"2024-04-22T08:34:15.967714+0000","last_clean":"2024-04-22T08:34:15.967714+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:15.967714+0000","last_undegraded":"2024-04-22T08:34:15.967714+0000","last_fullsized":"2024-04-22T08:34:15.967714+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647360,"num_objects":5285,"num_object_clones":0,"num_object_copies":10570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14711","reported_seq":15245,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.902478+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:15.902478+0000","last_peered":"2024-04-22T08:34:15.902478+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:15.902478+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102616,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11538,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'14915","reported_seq":14948,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.949408+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:15.949408+0000","last_peered":"2024-04-22T08:34:15.949408+0000","last_clean":"2024-04-22T08:34:15.949408+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:15.949408+0000","last_undegraded":"2024-04-22T08:34:15.949408+0000","last_fullsized":"2024-04-22T08:34:15.949408+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21295104,"num_objects":5199,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14776","reported_seq":14918,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.236745+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:16.236745+0000","last_peered":"2024-04-22T08:34:16.236745+0000","last_clean":"2024-04-22T08:34:16.236745+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:16.236745+0000","last_undegraded":"2024-04-22T08:34:16.236745+0000","last_fullsized":"2024-04-22T08:34:16.236745+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21065728,"num_objects":5143,"num_object_clones":0,"num_object_copies":10286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'14912","reported_seq":20705,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.147505+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:16.147505+0000","last_peered":"2024-04-22T08:34:16.147505+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:16.147505+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21032960,"num_objects":5135,"num_object_clones":0,"num_object_copies":10270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14699","reported_seq":22438,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.330555+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:15.330555+0000","last_peered":"2024-04-22T08:34:15.330555+0000","last_clean":"2024-04-22T08:34:15.330555+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:15.330555+0000","last_undegraded":"2024-04-22T08:34:15.330555+0000","last_fullsized":"2024-04-22T08:34:15.330555+0000","mapping_epoch":36,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21155840,"num_objects":5165,"num_object_clones":0,"num_object_copies":10330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14645","reported_seq":14792,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:16.280397+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:16.280397+0000","last_peered":"2024-04-22T08:34:16.280397+0000","last_clean":"2024-04-22T08:34:16.280397+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:16.280397+0000","last_undegraded":"2024-04-22T08:34:16.280397+0000","last_fullsized":"2024-04-22T08:34:16.280397+0000","mapping_epoch":27,"log_start":"35'4600","ondisk_log_start":"35'4600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21139456,"num_objects":5161,"num_object_clones":0,"num_object_copies":10322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15048","reported_seq":15505,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.889802+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:15.889802+0000","last_peered":"2024-04-22T08:34:15.889802+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:15.889802+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21189842,"num_objects":5175,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14988,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'14878","reported_seq":15756,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.298131+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:16.298131+0000","last_peered":"2024-04-22T08:34:16.298131+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:16.298131+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21742802,"num_objects":5310,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14876,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14728","reported_seq":15260,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:15.914578+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:15.914578+0000","last_peered":"2024-04-22T08:34:15.914578+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:15.914578+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21303296,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1642}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14802","reported_seq":22480,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:15.921662+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:15.921662+0000","last_peered":"2024-04-22T08:34:15.921662+0000","last_clean":"2024-04-22T08:34:15.921662+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:15.921662+0000","last_undegraded":"2024-04-22T08:34:15.921662+0000","last_fullsized":"2024-04-22T08:34:15.921662+0000","mapping_epoch":38,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21021906,"num_objects":5134,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14806","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.218914+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:16.218914+0000","last_peered":"2024-04-22T08:34:16.218914+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:16.218914+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21856873,"num_objects":5337,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14805,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14709","reported_seq":15381,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.256479+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:16.256479+0000","last_peered":"2024-04-22T08:34:16.256479+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:16.256479+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14794","reported_seq":15396,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.264681+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:16.264681+0000","last_peered":"2024-04-22T08:34:16.264681+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:16.264681+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21480658,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14792,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'14887","reported_seq":15601,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:16.255748+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:16.255748+0000","last_peered":"2024-04-22T08:34:16.255748+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:16.255748+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20953508,"num_objects":5119,"num_object_clones":0,"num_object_copies":10238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":340152173,"num_objects":83057,"num_object_clones":0,"num_object_copies":166114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210647,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12833,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867975168,"data_stored":867900923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160672,"ondisk_log_size":160672,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117083,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616484,"kb_used_data":287900,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93755356,"statfs":{"total":96636764160,"available":96005484544,"internally_reserved":0,"allocated":294809600,"data_stored":294583673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.2230000000000001,"15min":0.93000000000000005},"min":{"1min":0.217,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.859,"5min":13.635,"15min":13.635},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.137,"15min":0.73499999999999999},"min":{"1min":0.22,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":6.7670000000000003,"5min":13.721,"15min":13.721},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.248}]}]},{"osd":1,"up_from":33,"seq":141733920845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615424,"kb_used_data":250736,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93756416,"statfs":{"total":96636764160,"available":96006569984,"internally_reserved":0,"allocated":256753664,"data_stored":256472073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":373419724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.215}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.372}]}]},{"osd":0,"up_from":21,"seq":90194313311,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699280,"kb_used_data":310856,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93672560,"statfs":{"total":96636764160,"available":95920701440,"internally_reserved":0,"allocated":318316544,"data_stored":318101151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":11.368},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.2769999999999999,"15min":0.88600000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.3620000000000001,"5min":8,"15min":8},"last":0.216},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.02,"15min":0.85899999999999999},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317489152,"data_stored":317485673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256225280,"data_stored":256162065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294260736,"data_stored":294253185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:24.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:25.006 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:25.046 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-04-22T08:34:24.326913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330674029,"num_objects":80743,"num_object_clones":0,"num_object_copies":161486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212961,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1900352,"kb_used_data":830948,"kb_used_omap":12,"kb_used_meta":1069363,"kb_avail":281215168,"statfs":{"total":289910292480,"available":287964332032,"internally_reserved":0,"allocated":850890752,"data_stored":850167841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1095027822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,8,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18997248,"num_objects":-4638,"num_object_clones":0,"num_object_copies":-9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002797"},"pg_stats":[{"pgid":"2.f","version":"39'15061","reported_seq":15780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.191833+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:21.191833+0000","last_peered":"2024-04-22T08:34:21.191833+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:21.191833+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918272,"num_objects":5107,"num_object_clones":0,"num_object_copies":10214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15088","reported_seq":15206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.911202+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:20.911202+0000","last_peered":"2024-04-22T08:34:20.911202+0000","last_clean":"2024-04-22T08:34:20.911202+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:20.911202+0000","last_undegraded":"2024-04-22T08:34:20.911202+0000","last_fullsized":"2024-04-22T08:34:20.911202+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21069824,"num_objects":5144,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12144,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14862","reported_seq":15391,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.939560+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:20.939560+0000","last_peered":"2024-04-22T08:34:20.939560+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:20.939560+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20484120,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11689,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15073","reported_seq":15107,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.928765+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:20.928765+0000","last_peered":"2024-04-22T08:34:20.928765+0000","last_clean":"2024-04-22T08:34:20.928765+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:20.928765+0000","last_undegraded":"2024-04-22T08:34:20.928765+0000","last_fullsized":"2024-04-22T08:34:20.928765+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14919","reported_seq":15061,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.352221+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:21.352221+0000","last_peered":"2024-04-22T08:34:21.352221+0000","last_clean":"2024-04-22T08:34:21.352221+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:21.352221+0000","last_undegraded":"2024-04-22T08:34:21.352221+0000","last_fullsized":"2024-04-22T08:34:21.352221+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":5000,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15057","reported_seq":21080,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.251613+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:21.251613+0000","last_peered":"2024-04-22T08:34:21.251613+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:21.251613+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20439040,"num_objects":4990,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14824","reported_seq":22561,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.281916+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:20.281916+0000","last_peered":"2024-04-22T08:34:20.281916+0000","last_clean":"2024-04-22T08:34:20.281916+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:20.281916+0000","last_undegraded":"2024-04-22T08:34:20.281916+0000","last_fullsized":"2024-04-22T08:34:20.281916+0000","mapping_epoch":36,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":5040,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14799","reported_seq":14947,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.318905+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:21.318905+0000","last_peered":"2024-04-22T08:34:21.318905+0000","last_clean":"2024-04-22T08:34:21.318905+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:21.318905+0000","last_undegraded":"2024-04-22T08:34:21.318905+0000","last_fullsized":"2024-04-22T08:34:21.318905+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20508672,"num_objects":5007,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15202","reported_seq":15659,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.975217+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:20.975217+0000","last_peered":"2024-04-22T08:34:20.975217+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:20.975217+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20559058,"num_objects":5021,"num_object_clones":0,"num_object_copies":10042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15142,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1298}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15039","reported_seq":15916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.264445+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:21.264445+0000","last_peered":"2024-04-22T08:34:21.264445+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:21.264445+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21083346,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14888","reported_seq":15420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.927664+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:20.927664+0000","last_peered":"2024-04-22T08:34:20.927664+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:20.927664+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14934","reported_seq":22612,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.933912+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:20.933912+0000","last_peered":"2024-04-22T08:34:20.933912+0000","last_clean":"2024-04-22T08:34:20.933912+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:20.933912+0000","last_undegraded":"2024-04-22T08:34:20.933912+0000","last_fullsized":"2024-04-22T08:34:20.933912+0000","mapping_epoch":38,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481234,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14942","reported_seq":15697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.318239+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:21.318239+0000","last_peered":"2024-04-22T08:34:21.318239+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:21.318239+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299817,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14941,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1816}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14840","reported_seq":15512,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.362203+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:21.362203+0000","last_peered":"2024-04-22T08:34:21.362203+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:21.362203+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20111360,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14920","reported_seq":15522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.342547+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:21.342547+0000","last_peered":"2024-04-22T08:34:21.342547+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:21.342547+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20964562,"num_objects":5120,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15038","reported_seq":15752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.371118+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:21.371118+0000","last_peered":"2024-04-22T08:34:21.371118+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:21.371118+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20335012,"num_objects":4968,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":330674029,"num_objects":80743,"num_object_clones":0,"num_object_copies":161486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212961,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":848969728,"data_stored":848895483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117084,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599356,"kb_used_data":283060,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93772484,"statfs":{"total":96636764160,"available":96023023616,"internally_reserved":0,"allocated":289853440,"data_stored":289627513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.23999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.22700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920846,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600472,"kb_used_data":243980,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93771368,"statfs":{"total":96636764160,"available":96021880832,"internally_reserved":0,"allocated":249835520,"data_stored":249553929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.921},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.27400000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313312,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700524,"kb_used_data":303908,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93671316,"statfs":{"total":96636764160,"available":95919427584,"internally_reserved":0,"allocated":311201792,"data_stored":310986399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406122195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.437},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310513664,"data_stored":310510185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249131008,"data_stored":249067793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289325056,"data_stored":289317505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:25.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:25.703 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:25.704 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:25.740 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-04-22T08:34:24.326913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330674029,"num_objects":80743,"num_object_clones":0,"num_object_copies":161486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212961,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1900352,"kb_used_data":830948,"kb_used_omap":12,"kb_used_meta":1069363,"kb_avail":281215168,"statfs":{"total":289910292480,"available":287964332032,"internally_reserved":0,"allocated":850890752,"data_stored":850167841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1095027822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,8,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18997248,"num_objects":-4638,"num_object_clones":0,"num_object_copies":-9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002797"},"pg_stats":[{"pgid":"2.f","version":"39'15061","reported_seq":15780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.191833+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:21.191833+0000","last_peered":"2024-04-22T08:34:21.191833+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:21.191833+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918272,"num_objects":5107,"num_object_clones":0,"num_object_copies":10214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15088","reported_seq":15206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.911202+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:20.911202+0000","last_peered":"2024-04-22T08:34:20.911202+0000","last_clean":"2024-04-22T08:34:20.911202+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:20.911202+0000","last_undegraded":"2024-04-22T08:34:20.911202+0000","last_fullsized":"2024-04-22T08:34:20.911202+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21069824,"num_objects":5144,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12144,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'14862","reported_seq":15391,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.939560+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:20.939560+0000","last_peered":"2024-04-22T08:34:20.939560+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:20.939560+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20484120,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11689,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15073","reported_seq":15107,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.928765+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:20.928765+0000","last_peered":"2024-04-22T08:34:20.928765+0000","last_clean":"2024-04-22T08:34:20.928765+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:20.928765+0000","last_undegraded":"2024-04-22T08:34:20.928765+0000","last_fullsized":"2024-04-22T08:34:20.928765+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14919","reported_seq":15061,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.352221+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:21.352221+0000","last_peered":"2024-04-22T08:34:21.352221+0000","last_clean":"2024-04-22T08:34:21.352221+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:21.352221+0000","last_undegraded":"2024-04-22T08:34:21.352221+0000","last_fullsized":"2024-04-22T08:34:21.352221+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":5000,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15057","reported_seq":21080,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.251613+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:21.251613+0000","last_peered":"2024-04-22T08:34:21.251613+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:21.251613+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20439040,"num_objects":4990,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14824","reported_seq":22561,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.281916+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:20.281916+0000","last_peered":"2024-04-22T08:34:20.281916+0000","last_clean":"2024-04-22T08:34:20.281916+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:20.281916+0000","last_undegraded":"2024-04-22T08:34:20.281916+0000","last_fullsized":"2024-04-22T08:34:20.281916+0000","mapping_epoch":36,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":5040,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14799","reported_seq":14947,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.318905+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:21.318905+0000","last_peered":"2024-04-22T08:34:21.318905+0000","last_clean":"2024-04-22T08:34:21.318905+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:21.318905+0000","last_undegraded":"2024-04-22T08:34:21.318905+0000","last_fullsized":"2024-04-22T08:34:21.318905+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20508672,"num_objects":5007,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15202","reported_seq":15659,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.975217+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:20.975217+0000","last_peered":"2024-04-22T08:34:20.975217+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:20.975217+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20559058,"num_objects":5021,"num_object_clones":0,"num_object_copies":10042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15142,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1298}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15039","reported_seq":15916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.264445+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:21.264445+0000","last_peered":"2024-04-22T08:34:21.264445+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:21.264445+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21083346,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'14888","reported_seq":15420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:20.927664+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:20.927664+0000","last_peered":"2024-04-22T08:34:20.927664+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:20.927664+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20647936,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'14934","reported_seq":22612,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:20.933912+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:20.933912+0000","last_peered":"2024-04-22T08:34:20.933912+0000","last_clean":"2024-04-22T08:34:20.933912+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:20.933912+0000","last_undegraded":"2024-04-22T08:34:20.933912+0000","last_fullsized":"2024-04-22T08:34:20.933912+0000","mapping_epoch":38,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481234,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14942","reported_seq":15697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.318239+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:21.318239+0000","last_peered":"2024-04-22T08:34:21.318239+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:21.318239+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299817,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14941,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1816}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14840","reported_seq":15512,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.362203+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:21.362203+0000","last_peered":"2024-04-22T08:34:21.362203+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:21.362203+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20111360,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14920","reported_seq":15522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.342547+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:21.342547+0000","last_peered":"2024-04-22T08:34:21.342547+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:21.342547+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20964562,"num_objects":5120,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":58,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-22T08:32:18.133382+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:18.133382+0000","last_peered":"2024-04-22T08:32:18.133382+0000","last_clean":"2024-04-22T08:32:18.133382+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:18.133382+0000","last_undegraded":"2024-04-22T08:32:18.133382+0000","last_fullsized":"2024-04-22T08:32:18.133382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15038","reported_seq":15752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.371118+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:21.371118+0000","last_peered":"2024-04-22T08:34:21.371118+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:21.371118+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20335012,"num_objects":4968,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":330674029,"num_objects":80743,"num_object_clones":0,"num_object_copies":161486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212961,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":848969728,"data_stored":848895483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117084,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599356,"kb_used_data":283060,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93772484,"statfs":{"total":96636764160,"available":96023023616,"internally_reserved":0,"allocated":289853440,"data_stored":289627513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.23999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.22700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920846,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600472,"kb_used_data":243980,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93771368,"statfs":{"total":96636764160,"available":96021880832,"internally_reserved":0,"allocated":249835520,"data_stored":249553929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":365031116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.921},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.27400000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313312,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700524,"kb_used_data":303908,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93671316,"statfs":{"total":96636764160,"available":95919427584,"internally_reserved":0,"allocated":311201792,"data_stored":310986399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406122195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.437},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310513664,"data_stored":310510185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249131008,"data_stored":249067793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289325056,"data_stored":289317505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:25.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:25.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'15061", 'reported_seq': 15780, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.191833+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:21.191833+0000', 'last_peered': '2024-04-22T08:34:21.191833+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:21.191833+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20918272, 'num_objects': 5107, 'num_object_clones': 0, 'num_object_copies': 10214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15061, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1639}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:25.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'14862", 'reported_seq': 15391, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:20.939560+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:20.939560+0000', 'last_peered': '2024-04-22T08:34:20.939560+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:20.939560+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20484120, 'num_objects': 5002, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5002, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11689, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:25.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'15057", 'reported_seq': 21080, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.251613+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:21.251613+0000', 'last_peered': '2024-04-22T08:34:21.251613+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:21.251613+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20439040, 'num_objects': 4990, 'num_object_clones': 0, 'num_object_copies': 9980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11926, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2931, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'15202", 'reported_seq': 15659, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:20.975217+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:20.975217+0000', 'last_peered': '2024-04-22T08:34:20.975217+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:20.975217+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'5200", 'ondisk_log_start': "35'5200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20559058, 'num_objects': 5021, 'num_object_clones': 0, 'num_object_copies': 10042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15142, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'15039", 'reported_seq': 15916, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.264445+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:21.264445+0000', 'last_peered': '2024-04-22T08:34:21.264445+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:21.264445+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21083346, 'num_objects': 5149, 'num_object_clones': 0, 'num_object_copies': 10298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15037, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1778}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'14888", 'reported_seq': 15420, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:20.927664+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:20.927664+0000', 'last_peered': '2024-04-22T08:34:20.927664+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:20.927664+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20647936, 'num_objects': 5041, 'num_object_clones': 0, 'num_object_copies': 10082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11342, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1482}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'14942", 'reported_seq': 15697, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.318239+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:21.318239+0000', 'last_peered': '2024-04-22T08:34:21.318239+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:21.318239+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21299817, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 10402, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14941, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1816}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:25.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14840", 'reported_seq': 15512, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.362203+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:21.362203+0000', 'last_peered': '2024-04-22T08:34:21.362203+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:21.362203+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4800", 'ondisk_log_start': "35'4800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20111360, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 9820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11071, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:25.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'14920", 'reported_seq': 15522, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.342547+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:21.342547+0000', 'last_peered': '2024-04-22T08:34:21.342547+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:21.342547+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20964562, 'num_objects': 5120, 'num_object_clones': 0, 'num_object_copies': 10240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14918, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:25.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'15038", 'reported_seq': 15752, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:21.371118+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:21.371118+0000', 'last_peered': '2024-04-22T08:34:21.371118+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:21.371118+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20335012, 'num_objects': 4968, 'num_object_clones': 0, 'num_object_copies': 9936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15035, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:25.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:26.820 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:26.821 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32657/159490 objects degraded (20.476%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":501,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":79745,"data_bytes":326586221,"bytes_used":1942511616,"bytes_avail":287967780864,"bytes_total":289910292480,"degraded_objects":32657,"degraded_total":159490,"degraded_ratio":0.20475891905448618,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":469},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:26.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:26.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:27.433 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:27.433 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:27.466 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-04-22T08:34:26.327486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":326586221,"num_objects":79745,"num_object_clones":0,"num_object_copies":159490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213959,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1896984,"kb_used_data":819388,"kb_used_omap":12,"kb_used_meta":1077555,"kb_avail":281218536,"statfs":{"total":289910292480,"available":287967780864,"internally_reserved":0,"allocated":839053312,"data_stored":838330401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1103416430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3,11,6,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23085056,"num_objects":-5636,"num_object_clones":0,"num_object_copies":-11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.f","version":"39'15061","reported_seq":15780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.191833+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:21.191833+0000","last_peered":"2024-04-22T08:34:21.191833+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:21.191833+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918272,"num_objects":5107,"num_object_clones":0,"num_object_copies":10214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15220","reported_seq":15337,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.032434+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:26.032434+0000","last_peered":"2024-04-22T08:34:26.032434+0000","last_clean":"2024-04-22T08:34:26.032434+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:26.032434+0000","last_undegraded":"2024-04-22T08:34:26.032434+0000","last_fullsized":"2024-04-22T08:34:26.032434+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20529152,"num_objects":5012,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15030","reported_seq":15564,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.989184+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:25.989184+0000","last_peered":"2024-04-22T08:34:25.989184+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:25.989184+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19795992,"num_objects":4834,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11857,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15219","reported_seq":15253,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.917313+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:25.917313+0000","last_peered":"2024-04-22T08:34:25.917313+0000","last_clean":"2024-04-22T08:34:25.917313+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:25.917313+0000","last_undegraded":"2024-04-22T08:34:25.917313+0000","last_fullsized":"2024-04-22T08:34:25.917313+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20049920,"num_objects":4895,"num_object_clones":0,"num_object_copies":9790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15219,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'14919","reported_seq":15061,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.352221+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:21.352221+0000","last_peered":"2024-04-22T08:34:21.352221+0000","last_clean":"2024-04-22T08:34:21.352221+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:21.352221+0000","last_undegraded":"2024-04-22T08:34:21.352221+0000","last_fullsized":"2024-04-22T08:34:21.352221+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":5000,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15057","reported_seq":21080,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.251613+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:21.251613+0000","last_peered":"2024-04-22T08:34:21.251613+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:21.251613+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20439040,"num_objects":4990,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14948","reported_seq":22686,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.335994+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:25.335994+0000","last_peered":"2024-04-22T08:34:25.335994+0000","last_clean":"2024-04-22T08:34:25.335994+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:25.335994+0000","last_undegraded":"2024-04-22T08:34:25.335994+0000","last_fullsized":"2024-04-22T08:34:25.335994+0000","mapping_epoch":36,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20135936,"num_objects":4916,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14799","reported_seq":14947,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:21.318905+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:21.318905+0000","last_peered":"2024-04-22T08:34:21.318905+0000","last_clean":"2024-04-22T08:34:21.318905+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:21.318905+0000","last_undegraded":"2024-04-22T08:34:21.318905+0000","last_fullsized":"2024-04-22T08:34:21.318905+0000","mapping_epoch":27,"log_start":"35'4700","ondisk_log_start":"35'4700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20508672,"num_objects":5007,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15333","reported_seq":15790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.946240+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:25.946240+0000","last_peered":"2024-04-22T08:34:25.946240+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:25.946240+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20022482,"num_objects":4890,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15039","reported_seq":15916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.264445+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:21.264445+0000","last_peered":"2024-04-22T08:34:21.264445+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:21.264445+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21083346,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15029","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.995086+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:25.995086+0000","last_peered":"2024-04-22T08:34:25.995086+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:25.995086+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20070400,"num_objects":4900,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15090","reported_seq":22767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.974429+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:25.974429+0000","last_peered":"2024-04-22T08:34:25.974429+0000","last_clean":"2024-04-22T08:34:25.974429+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:25.974429+0000","last_undegraded":"2024-04-22T08:34:25.974429+0000","last_fullsized":"2024-04-22T08:34:25.974429+0000","mapping_epoch":38,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19842258,"num_objects":4846,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15088,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'14942","reported_seq":15697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.318239+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:21.318239+0000","last_peered":"2024-04-22T08:34:21.318239+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:21.318239+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299817,"num_objects":5201,"num_object_clones":0,"num_object_copies":10402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14941,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1816}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14840","reported_seq":15512,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.362203+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:21.362203+0000","last_peered":"2024-04-22T08:34:21.362203+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:21.362203+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4800","ondisk_log_start":"35'4800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20111360,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'14920","reported_seq":15522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.342547+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:21.342547+0000","last_peered":"2024-04-22T08:34:21.342547+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:21.342547+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20964562,"num_objects":5120,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15038","reported_seq":15752,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:21.371118+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:21.371118+0000","last_peered":"2024-04-22T08:34:21.371118+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:21.371118+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20335012,"num_objects":4968,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":326586221,"num_objects":79745,"num_object_clones":0,"num_object_copies":159490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213959,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12950,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":836911104,"data_stored":836836859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117085,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598868,"kb_used_data":278472,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93772972,"statfs":{"total":96636764160,"available":96023523328,"internally_reserved":0,"allocated":285155328,"data_stored":284929401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":3.4369999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":3.4630000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":3.2469999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597592,"kb_used_data":237008,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93774248,"statfs":{"total":96636764160,"available":96024829952,"internally_reserved":0,"allocated":242696192,"data_stored":242414601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":5.3120000000000003},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":5.0869999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.29699999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313312,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700524,"kb_used_data":303908,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93671316,"statfs":{"total":96636764160,"available":95919427584,"internally_reserved":0,"allocated":311201792,"data_stored":310986399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406122195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.437},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310513664,"data_stored":310510185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241819648,"data_stored":241756433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284577792,"data_stored":284570241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:30.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:30.871 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:30.901 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-04-22T08:34:30.328787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320986989,"num_objects":78378,"num_object_clones":0,"num_object_copies":156756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215326,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13066,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1894292,"kb_used_data":812600,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281221228,"statfs":{"total":289910292480,"available":287970537472,"internally_reserved":0,"allocated":832102400,"data_stored":831379489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,15,6,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19165184,"num_objects":-4679,"num_object_clones":0,"num_object_copies":-9358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4679,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"2.f","version":"39'15218","reported_seq":15937,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.340781+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:26.340781+0000","last_peered":"2024-04-22T08:34:26.340781+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:26.340781+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20275200,"num_objects":4950,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15218,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1482}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15220","reported_seq":15337,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.032434+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:26.032434+0000","last_peered":"2024-04-22T08:34:26.032434+0000","last_clean":"2024-04-22T08:34:26.032434+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:26.032434+0000","last_undegraded":"2024-04-22T08:34:26.032434+0000","last_fullsized":"2024-04-22T08:34:26.032434+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20529152,"num_objects":5012,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15030","reported_seq":15564,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.989184+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:25.989184+0000","last_peered":"2024-04-22T08:34:25.989184+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:25.989184+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19795992,"num_objects":4834,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11857,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15219","reported_seq":15253,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.917313+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:25.917313+0000","last_peered":"2024-04-22T08:34:25.917313+0000","last_clean":"2024-04-22T08:34:25.917313+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:25.917313+0000","last_undegraded":"2024-04-22T08:34:25.917313+0000","last_fullsized":"2024-04-22T08:34:25.917313+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20049920,"num_objects":4895,"num_object_clones":0,"num_object_copies":9790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15219,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15068","reported_seq":15210,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.305351+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:26.305351+0000","last_peered":"2024-04-22T08:34:26.305351+0000","last_clean":"2024-04-22T08:34:26.305351+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:26.305351+0000","last_undegraded":"2024-04-22T08:34:26.305351+0000","last_fullsized":"2024-04-22T08:34:26.305351+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19869696,"num_objects":4851,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15200","reported_seq":21459,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.374027+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:26.374027+0000","last_peered":"2024-04-22T08:34:26.374027+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:26.374027+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14948","reported_seq":22686,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.335994+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:25.335994+0000","last_peered":"2024-04-22T08:34:25.335994+0000","last_clean":"2024-04-22T08:34:25.335994+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:25.335994+0000","last_undegraded":"2024-04-22T08:34:25.335994+0000","last_fullsized":"2024-04-22T08:34:25.335994+0000","mapping_epoch":36,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20135936,"num_objects":4916,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14948","reported_seq":15096,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.376998+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:26.376998+0000","last_peered":"2024-04-22T08:34:26.376998+0000","last_clean":"2024-04-22T08:34:26.376998+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:26.376998+0000","last_undegraded":"2024-04-22T08:34:26.376998+0000","last_fullsized":"2024-04-22T08:34:26.376998+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":4858,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15333","reported_seq":15790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.946240+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:25.946240+0000","last_peered":"2024-04-22T08:34:25.946240+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:25.946240+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20022482,"num_objects":4890,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15186","reported_seq":16067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.299071+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:26.299071+0000","last_peered":"2024-04-22T08:34:26.299071+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:26.299071+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481234,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1631}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15029","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.995086+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:25.995086+0000","last_peered":"2024-04-22T08:34:25.995086+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:25.995086+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20070400,"num_objects":4900,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15090","reported_seq":22767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.974429+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:25.974429+0000","last_peered":"2024-04-22T08:34:25.974429+0000","last_clean":"2024-04-22T08:34:25.974429+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:25.974429+0000","last_undegraded":"2024-04-22T08:34:25.974429+0000","last_fullsized":"2024-04-22T08:34:25.974429+0000","mapping_epoch":38,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19842258,"num_objects":4846,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15088,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15104","reported_seq":15859,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.253801+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:26.253801+0000","last_peered":"2024-04-22T08:34:26.253801+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:26.253801+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20636265,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14991","reported_seq":15664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.368927+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:26.368927+0000","last_peered":"2024-04-22T08:34:26.368927+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:26.368927+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19492864,"num_objects":4759,"num_object_clones":0,"num_object_copies":9518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15085","reported_seq":15683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.410384+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:26.410384+0000","last_peered":"2024-04-22T08:34:26.410384+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:26.410384+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20288722,"num_objects":4955,"num_object_clones":0,"num_object_copies":9910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15083,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15182","reported_seq":15896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.399065+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:26.399065+0000","last_peered":"2024-04-22T08:34:26.399065+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:26.399065+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19745188,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15179,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":320986989,"num_objects":78378,"num_object_clones":0,"num_object_copies":156756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215326,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13066,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829648896,"data_stored":829574651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117085,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598868,"kb_used_data":278472,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93772972,"statfs":{"total":96636764160,"available":96023523328,"internally_reserved":0,"allocated":285155328,"data_stored":284929401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":3.4369999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":3.4630000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":3.2469999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597592,"kb_used_data":237008,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93774248,"statfs":{"total":96636764160,"available":96024829952,"internally_reserved":0,"allocated":242696192,"data_stored":242414601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":5.3120000000000003},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":5.0869999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.29699999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313313,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697832,"kb_used_data":297120,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93674008,"statfs":{"total":96636764160,"available":95922184192,"internally_reserved":0,"allocated":304250880,"data_stored":304035487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303251456,"data_stored":303247977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241819648,"data_stored":241756433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284577792,"data_stored":284570241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:30.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:31.683 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:31.685 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:31.718 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-04-22T08:34:30.328787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320986989,"num_objects":78378,"num_object_clones":0,"num_object_copies":156756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215326,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13066,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1894292,"kb_used_data":812600,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281221228,"statfs":{"total":289910292480,"available":287970537472,"internally_reserved":0,"allocated":832102400,"data_stored":831379489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1107610734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,15,6,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19165184,"num_objects":-4679,"num_object_clones":0,"num_object_copies":-9358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4679,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"2.f","version":"39'15218","reported_seq":15937,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.340781+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:26.340781+0000","last_peered":"2024-04-22T08:34:26.340781+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:26.340781+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20275200,"num_objects":4950,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15218,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1482}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15220","reported_seq":15337,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.032434+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:26.032434+0000","last_peered":"2024-04-22T08:34:26.032434+0000","last_clean":"2024-04-22T08:34:26.032434+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:26.032434+0000","last_undegraded":"2024-04-22T08:34:26.032434+0000","last_fullsized":"2024-04-22T08:34:26.032434+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20529152,"num_objects":5012,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15030","reported_seq":15564,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.989184+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:25.989184+0000","last_peered":"2024-04-22T08:34:25.989184+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:25.989184+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19795992,"num_objects":4834,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11857,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15219","reported_seq":15253,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.917313+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:25.917313+0000","last_peered":"2024-04-22T08:34:25.917313+0000","last_clean":"2024-04-22T08:34:25.917313+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:25.917313+0000","last_undegraded":"2024-04-22T08:34:25.917313+0000","last_fullsized":"2024-04-22T08:34:25.917313+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20049920,"num_objects":4895,"num_object_clones":0,"num_object_copies":9790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15219,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15068","reported_seq":15210,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.305351+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:26.305351+0000","last_peered":"2024-04-22T08:34:26.305351+0000","last_clean":"2024-04-22T08:34:26.305351+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:26.305351+0000","last_undegraded":"2024-04-22T08:34:26.305351+0000","last_fullsized":"2024-04-22T08:34:26.305351+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19869696,"num_objects":4851,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15200","reported_seq":21459,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.374027+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:26.374027+0000","last_peered":"2024-04-22T08:34:26.374027+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:26.374027+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'14948","reported_seq":22686,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.335994+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:25.335994+0000","last_peered":"2024-04-22T08:34:25.335994+0000","last_clean":"2024-04-22T08:34:25.335994+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:25.335994+0000","last_undegraded":"2024-04-22T08:34:25.335994+0000","last_fullsized":"2024-04-22T08:34:25.335994+0000","mapping_epoch":36,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20135936,"num_objects":4916,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'14948","reported_seq":15096,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:26.376998+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:26.376998+0000","last_peered":"2024-04-22T08:34:26.376998+0000","last_clean":"2024-04-22T08:34:26.376998+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:26.376998+0000","last_undegraded":"2024-04-22T08:34:26.376998+0000","last_fullsized":"2024-04-22T08:34:26.376998+0000","mapping_epoch":27,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19898368,"num_objects":4858,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15333","reported_seq":15790,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.946240+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:25.946240+0000","last_peered":"2024-04-22T08:34:25.946240+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:25.946240+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20022482,"num_objects":4890,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15186","reported_seq":16067,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.299071+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:26.299071+0000","last_peered":"2024-04-22T08:34:26.299071+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:26.299071+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481234,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1631}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15029","reported_seq":15561,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:25.995086+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:25.995086+0000","last_peered":"2024-04-22T08:34:25.995086+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:25.995086+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20070400,"num_objects":4900,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15090","reported_seq":22767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:25.974429+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:25.974429+0000","last_peered":"2024-04-22T08:34:25.974429+0000","last_clean":"2024-04-22T08:34:25.974429+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:25.974429+0000","last_undegraded":"2024-04-22T08:34:25.974429+0000","last_fullsized":"2024-04-22T08:34:25.974429+0000","mapping_epoch":38,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19842258,"num_objects":4846,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15088,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15104","reported_seq":15859,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.253801+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:26.253801+0000","last_peered":"2024-04-22T08:34:26.253801+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:26.253801+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20636265,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'14991","reported_seq":15664,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.368927+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:26.368927+0000","last_peered":"2024-04-22T08:34:26.368927+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:26.368927+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'4900","ondisk_log_start":"35'4900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19492864,"num_objects":4759,"num_object_clones":0,"num_object_copies":9518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15085","reported_seq":15683,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.410384+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:26.410384+0000","last_peered":"2024-04-22T08:34:26.410384+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:26.410384+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20288722,"num_objects":4955,"num_object_clones":0,"num_object_copies":9910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15083,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15182","reported_seq":15896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:26.399065+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:26.399065+0000","last_peered":"2024-04-22T08:34:26.399065+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:26.399065+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19745188,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15179,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":320986989,"num_objects":78378,"num_object_clones":0,"num_object_copies":156756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215326,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13066,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829648896,"data_stored":829574651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117085,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598868,"kb_used_data":278472,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93772972,"statfs":{"total":96636764160,"available":96023523328,"internally_reserved":0,"allocated":285155328,"data_stored":284929401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":3.4369999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":3.4630000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":3.2469999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597592,"kb_used_data":237008,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93774248,"statfs":{"total":96636764160,"available":96024829952,"internally_reserved":0,"allocated":242696192,"data_stored":242414601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":5.3120000000000003},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":5.0869999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.29699999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313313,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697832,"kb_used_data":297120,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93674008,"statfs":{"total":96636764160,"available":95922184192,"internally_reserved":0,"allocated":304250880,"data_stored":304035487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303251456,"data_stored":303247977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241819648,"data_stored":241756433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284577792,"data_stored":284570241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:31.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'15218", 'reported_seq': 15937, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.340781+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:26.340781+0000', 'last_peered': '2024-04-22T08:34:26.340781+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:26.340781+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'5200", 'ondisk_log_start': "35'5200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20275200, 'num_objects': 4950, 'num_object_clones': 0, 'num_object_copies': 9900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15218, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1482}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15030", 'reported_seq': 15564, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:25.989184+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:25.989184+0000', 'last_peered': '2024-04-22T08:34:25.989184+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:25.989184+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19795992, 'num_objects': 4834, 'num_object_clones': 0, 'num_object_copies': 9668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4834, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11857, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'15200", 'reported_seq': 21459, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.374027+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:26.374027+0000', 'last_peered': '2024-04-22T08:34:26.374027+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:26.374027+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'5100", 'ondisk_log_start': "35'5100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19853312, 'num_objects': 4847, 'num_object_clones': 0, 'num_object_copies': 9694, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 835, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12069, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3047, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4012}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'15333", 'reported_seq': 15790, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:25.946240+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:25.946240+0000', 'last_peered': '2024-04-22T08:34:25.946240+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:25.946240+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'5300", 'ondisk_log_start': "35'5300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20022482, 'num_objects': 4890, 'num_object_clones': 0, 'num_object_copies': 9780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15273, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:31.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'15186", 'reported_seq': 16067, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.299071+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:26.299071+0000', 'last_peered': '2024-04-22T08:34:26.299071+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:26.299071+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'5100", 'ondisk_log_start': "35'5100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20481234, 'num_objects': 5002, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1631}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'15029", 'reported_seq': 15561, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:25.995086+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:25.995086+0000', 'last_peered': '2024-04-22T08:34:25.995086+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:25.995086+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20070400, 'num_objects': 4900, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11483, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1341}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'15104", 'reported_seq': 15859, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.253801+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:26.253801+0000', 'last_peered': '2024-04-22T08:34:26.253801+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:26.253801+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'5100", 'ondisk_log_start': "35'5100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20636265, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 10078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15103, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'14991", 'reported_seq': 15664, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.368927+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:26.368927+0000', 'last_peered': '2024-04-22T08:34:26.368927+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:26.368927+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'4900", 'ondisk_log_start': "35'4900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19492864, 'num_objects': 4759, 'num_object_clones': 0, 'num_object_copies': 9518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11222, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'15085", 'reported_seq': 15683, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.410384+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:26.410384+0000', 'last_peered': '2024-04-22T08:34:26.410384+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:26.410384+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'5000", 'ondisk_log_start': "35'5000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20288722, 'num_objects': 4955, 'num_object_clones': 0, 'num_object_copies': 9910, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15083, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:31.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'15182", 'reported_seq': 15896, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:26.399065+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:26.399065+0000', 'last_peered': '2024-04-22T08:34:26.399065+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:26.399065+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'5100", 'ondisk_log_start': "35'5100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19745188, 'num_objects': 4824, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15179, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:31.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:32.830 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:32.830 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32408/152300 objects degraded (21.279%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":507,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":76150,"data_bytes":311861101,"bytes_used":1908400128,"bytes_avail":288001892352,"bytes_total":289910292480,"degraded_objects":32408,"degraded_total":152300,"degraded_ratio":0.21279054497701902,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":366,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":575},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:32.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:32.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:33.572 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:33.673 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-22T08:34:32.329238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":311861101,"num_objects":76150,"num_object_clones":0,"num_object_copies":152300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76150,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217554,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13199,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1863672,"kb_used_data":794268,"kb_used_omap":12,"kb_used_meta":1069363,"kb_avail":281251848,"statfs":{"total":289910292480,"available":288001892352,"internally_reserved":0,"allocated":813330432,"data_stored":812607521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1095027822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,6,7,18,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28291072,"num_objects":-6907,"num_object_clones":0,"num_object_copies":-13814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6907,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003257"},"pg_stats":[{"pgid":"2.f","version":"39'15360","reported_seq":16077,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.392112+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:31.392112+0000","last_peered":"2024-04-22T08:34:31.392112+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:31.392112+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19693568,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15360,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15361","reported_seq":15480,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:31.006195+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:31.006195+0000","last_peered":"2024-04-22T08:34:31.006195+0000","last_clean":"2024-04-22T08:34:31.006195+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:31.006195+0000","last_undegraded":"2024-04-22T08:34:31.006195+0000","last_fullsized":"2024-04-22T08:34:31.006195+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19951616,"num_objects":4871,"num_object_clones":0,"num_object_copies":9742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12417,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15155","reported_seq":15688,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:30.966917+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:30.966917+0000","last_peered":"2024-04-22T08:34:30.966917+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:30.966917+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19283992,"num_objects":4709,"num_object_clones":0,"num_object_copies":9418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11982,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15344","reported_seq":15378,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:30.650330+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:30.650330+0000","last_peered":"2024-04-22T08:34:30.650330+0000","last_clean":"2024-04-22T08:34:30.650330+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:30.650330+0000","last_undegraded":"2024-04-22T08:34:30.650330+0000","last_fullsized":"2024-04-22T08:34:30.650330+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19537920,"num_objects":4770,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15344,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15203","reported_seq":15345,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:31.447978+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:31.447978+0000","last_peered":"2024-04-22T08:34:31.447978+0000","last_clean":"2024-04-22T08:34:31.447978+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:31.447978+0000","last_undegraded":"2024-04-22T08:34:31.447978+0000","last_fullsized":"2024-04-22T08:34:31.447978+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19316736,"num_objects":4716,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12295,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15355","reported_seq":21880,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.356826+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:31.356826+0000","last_peered":"2024-04-22T08:34:31.356826+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:31.356826+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218432,"num_objects":4692,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3180,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15096","reported_seq":22835,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:30.510380+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:30.510380+0000","last_peered":"2024-04-22T08:34:30.510380+0000","last_clean":"2024-04-22T08:34:30.510380+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:30.510380+0000","last_undegraded":"2024-04-22T08:34:30.510380+0000","last_fullsized":"2024-04-22T08:34:30.510380+0000","mapping_epoch":36,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":4768,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11464,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15076","reported_seq":15224,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:31.383917+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:31.383917+0000","last_peered":"2024-04-22T08:34:31.383917+0000","last_clean":"2024-04-22T08:34:31.383917+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:31.383917+0000","last_undegraded":"2024-04-22T08:34:31.383917+0000","last_fullsized":"2024-04-22T08:34:31.383917+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19374080,"num_objects":4730,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15462","reported_seq":15919,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:30.923688+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:30.923688+0000","last_peered":"2024-04-22T08:34:30.923688+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:30.923688+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19494098,"num_objects":4761,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15402,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15329","reported_seq":16210,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.324906+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:31.324906+0000","last_peered":"2024-04-22T08:34:31.324906+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:31.324906+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15327,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1488}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15174","reported_seq":15706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.014538+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:31.014538+0000","last_peered":"2024-04-22T08:34:31.014538+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:31.014538+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19476480,"num_objects":4755,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11628,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15229","reported_seq":22907,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:30.931261+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:30.931261+0000","last_peered":"2024-04-22T08:34:30.931261+0000","last_clean":"2024-04-22T08:34:30.931261+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:30.931261+0000","last_undegraded":"2024-04-22T08:34:30.931261+0000","last_fullsized":"2024-04-22T08:34:30.931261+0000","mapping_epoch":38,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19272914,"num_objects":4707,"num_object_clones":0,"num_object_copies":9414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15227,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15233","reported_seq":15984,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.384236+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:31.384236+0000","last_peered":"2024-04-22T08:34:31.384236+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:31.384236+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20107881,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15232,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15138","reported_seq":15811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.319753+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:31.319753+0000","last_peered":"2024-04-22T08:34:31.319753+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:31.319753+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18890752,"num_objects":4612,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15240","reported_seq":15842,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.406129+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:31.406129+0000","last_peered":"2024-04-22T08:34:31.406129+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:31.406129+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19653842,"num_objects":4800,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15238,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15324","reported_seq":16038,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.320064+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:31.320064+0000","last_peered":"2024-04-22T08:34:31.320064+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:31.320064+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19163556,"num_objects":4682,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15321,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":311861101,"num_objects":76150,"num_object_clones":0,"num_object_copies":152300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76150,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217554,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13199,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":810954752,"data_stored":810880507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117086,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598544,"kb_used_data":274056,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93773296,"statfs":{"total":96636764160,"available":96023855104,"internally_reserved":0,"allocated":280633344,"data_stored":280407417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":3.3130000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.249}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.224}]}]},{"osd":1,"up_from":33,"seq":141733920848,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582360,"kb_used_data":229964,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93789480,"statfs":{"total":96636764160,"available":96040427520,"internally_reserved":0,"allocated":235483136,"data_stored":235201545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":360836812},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":9.0660000000000007},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":1.9710000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":2.1960000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313314,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682768,"kb_used_data":290248,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93689072,"statfs":{"total":96636764160,"available":95937609728,"internally_reserved":0,"allocated":297213952,"data_stored":296998559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,6,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.6160000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296325120,"data_stored":296321641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234467328,"data_stored":234404113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280162304,"data_stored":280154753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:36.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:37.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:37.495 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:37.556 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-22T08:34:36.333951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":307679085,"num_objects":75129,"num_object_clones":0,"num_object_copies":150258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218575,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13199,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160500,"ondisk_log_size":160500,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1868728,"kb_used_data":782944,"kb_used_omap":12,"kb_used_meta":1085747,"kb_avail":281246792,"statfs":{"total":289910292480,"available":287996715008,"internally_reserved":0,"allocated":801734656,"data_stored":801011745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1111805038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,9,10,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22994944,"num_objects":-5614,"num_object_clones":0,"num_object_copies":-11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007037"},"pg_stats":[{"pgid":"2.f","version":"39'15360","reported_seq":16077,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.392112+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:31.392112+0000","last_peered":"2024-04-22T08:34:31.392112+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:31.392112+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19693568,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15360,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15516","reported_seq":15635,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.995667+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:35.995667+0000","last_peered":"2024-04-22T08:34:35.995667+0000","last_clean":"2024-04-22T08:34:35.995667+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:35.995667+0000","last_undegraded":"2024-04-22T08:34:35.995667+0000","last_fullsized":"2024-04-22T08:34:35.995667+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19316736,"num_objects":4716,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12572,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15307","reported_seq":15841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.880157+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:35.880157+0000","last_peered":"2024-04-22T08:34:35.880157+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:35.880157+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661400,"num_objects":4557,"num_object_clones":0,"num_object_copies":9114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12134,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15503","reported_seq":15537,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.996261+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:35.996261+0000","last_peered":"2024-04-22T08:34:35.996261+0000","last_clean":"2024-04-22T08:34:35.996261+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:35.996261+0000","last_undegraded":"2024-04-22T08:34:35.996261+0000","last_fullsized":"2024-04-22T08:34:35.996261+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15503,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15203","reported_seq":15345,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:31.447978+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:31.447978+0000","last_peered":"2024-04-22T08:34:31.447978+0000","last_clean":"2024-04-22T08:34:31.447978+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:31.447978+0000","last_undegraded":"2024-04-22T08:34:31.447978+0000","last_fullsized":"2024-04-22T08:34:31.447978+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19316736,"num_objects":4716,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12295,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15355","reported_seq":21880,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.356826+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:31.356826+0000","last_peered":"2024-04-22T08:34:31.356826+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:31.356826+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218432,"num_objects":4692,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3180,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15231","reported_seq":22971,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.558945+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:35.558945+0000","last_peered":"2024-04-22T08:34:35.558945+0000","last_clean":"2024-04-22T08:34:35.558945+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:35.558945+0000","last_undegraded":"2024-04-22T08:34:35.558945+0000","last_fullsized":"2024-04-22T08:34:35.558945+0000","mapping_epoch":36,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15076","reported_seq":15224,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:31.383917+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:31.383917+0000","last_peered":"2024-04-22T08:34:31.383917+0000","last_clean":"2024-04-22T08:34:31.383917+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:31.383917+0000","last_undegraded":"2024-04-22T08:34:31.383917+0000","last_fullsized":"2024-04-22T08:34:31.383917+0000","mapping_epoch":27,"log_start":"35'5000","ondisk_log_start":"35'5000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19374080,"num_objects":4730,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15606","reported_seq":16062,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.044788+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:36.044788+0000","last_peered":"2024-04-22T08:34:36.044788+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:36.044788+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904274,"num_objects":4617,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15546,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15329","reported_seq":16210,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.324906+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:31.324906+0000","last_peered":"2024-04-22T08:34:31.324906+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:31.324906+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15327,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1488}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15328","reported_seq":15860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.978690+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:35.978690+0000","last_peered":"2024-04-22T08:34:35.978690+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:35.978690+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18845696,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15351","reported_seq":23029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.979203+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:35.979203+0000","last_peered":"2024-04-22T08:34:35.979203+0000","last_clean":"2024-04-22T08:34:35.979203+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:35.979203+0000","last_undegraded":"2024-04-22T08:34:35.979203+0000","last_fullsized":"2024-04-22T08:34:35.979203+0000","mapping_epoch":38,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773202,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15349,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15233","reported_seq":15984,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.384236+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:31.384236+0000","last_peered":"2024-04-22T08:34:31.384236+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:31.384236+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20107881,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15232,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15138","reported_seq":15811,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.319753+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:31.319753+0000","last_peered":"2024-04-22T08:34:31.319753+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:31.319753+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5100","ondisk_log_start":"35'5100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18890752,"num_objects":4612,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15240","reported_seq":15842,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.406129+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:31.406129+0000","last_peered":"2024-04-22T08:34:31.406129+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:31.406129+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19653842,"num_objects":4800,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15238,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15324","reported_seq":16038,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:31.320064+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:31.320064+0000","last_peered":"2024-04-22T08:34:31.320064+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:31.320064+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19163556,"num_objects":4682,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15321,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":307679085,"num_objects":75129,"num_object_clones":0,"num_object_copies":150258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218575,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13199,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799133696,"data_stored":799059451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160500,"ondisk_log_size":160500,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117087,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601972,"kb_used_data":269292,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93769868,"statfs":{"total":96636764160,"available":96020344832,"internally_reserved":0,"allocated":275755008,"data_stored":275529081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.255}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920849,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583988,"kb_used_data":223404,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93787852,"statfs":{"total":96636764160,"available":96038760448,"internally_reserved":0,"allocated":228765696,"data_stored":228484105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.44400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313314,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682768,"kb_used_data":290248,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93689072,"statfs":{"total":96636764160,"available":95937609728,"internally_reserved":0,"allocated":297213952,"data_stored":296998559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,6,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.6160000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296325120,"data_stored":296321641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":227483648,"data_stored":227420433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275324928,"data_stored":275317377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:37.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:38.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:38.872 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:38.922 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-04-22T08:34:38.334464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302137197,"num_objects":73776,"num_object_clones":0,"num_object_copies":147552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219928,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13294,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160553,"ondisk_log_size":160553,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1865632,"kb_used_data":775752,"kb_used_omap":12,"kb_used_meta":1089843,"kb_avail":281249888,"statfs":{"total":289910292480,"available":287999885312,"internally_reserved":0,"allocated":794370048,"data_stored":793647137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1115999342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,7,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24449024,"num_objects":-5969,"num_object_clones":0,"num_object_copies":-11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006978"},"pg_stats":[{"pgid":"2.f","version":"39'15525","reported_seq":16244,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.463593+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:36.463593+0000","last_peered":"2024-04-22T08:34:36.463593+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:36.463593+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19017728,"num_objects":4643,"num_object_clones":0,"num_object_copies":9286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15525,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15516","reported_seq":15635,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.995667+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:35.995667+0000","last_peered":"2024-04-22T08:34:35.995667+0000","last_clean":"2024-04-22T08:34:35.995667+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:35.995667+0000","last_undegraded":"2024-04-22T08:34:35.995667+0000","last_fullsized":"2024-04-22T08:34:35.995667+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19316736,"num_objects":4716,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12572,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15307","reported_seq":15841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.880157+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:35.880157+0000","last_peered":"2024-04-22T08:34:35.880157+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:35.880157+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661400,"num_objects":4557,"num_object_clones":0,"num_object_copies":9114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12134,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15503","reported_seq":15537,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.996261+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:35.996261+0000","last_peered":"2024-04-22T08:34:35.996261+0000","last_clean":"2024-04-22T08:34:35.996261+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:35.996261+0000","last_undegraded":"2024-04-22T08:34:35.996261+0000","last_fullsized":"2024-04-22T08:34:35.996261+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15503,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15345","reported_seq":15487,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:36.343841+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:36.343841+0000","last_peered":"2024-04-22T08:34:36.343841+0000","last_clean":"2024-04-22T08:34:36.343841+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:36.343841+0000","last_undegraded":"2024-04-22T08:34:36.343841+0000","last_fullsized":"2024-04-22T08:34:36.343841+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18735104,"num_objects":4574,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12437,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15514","reported_seq":22229,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.473477+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:36.473477+0000","last_peered":"2024-04-22T08:34:36.473477+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:36.473477+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18567168,"num_objects":4533,"num_object_clones":0,"num_object_copies":9066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3926}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15231","reported_seq":22971,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.558945+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:35.558945+0000","last_peered":"2024-04-22T08:34:35.558945+0000","last_clean":"2024-04-22T08:34:35.558945+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:35.558945+0000","last_undegraded":"2024-04-22T08:34:35.558945+0000","last_fullsized":"2024-04-22T08:34:35.558945+0000","mapping_epoch":36,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15234","reported_seq":15382,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:36.405966+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:36.405966+0000","last_peered":"2024-04-22T08:34:36.405966+0000","last_clean":"2024-04-22T08:34:36.405966+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:36.405966+0000","last_undegraded":"2024-04-22T08:34:36.405966+0000","last_fullsized":"2024-04-22T08:34:36.405966+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18726912,"num_objects":4572,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15606","reported_seq":16062,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.044788+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:36.044788+0000","last_peered":"2024-04-22T08:34:36.044788+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:36.044788+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904274,"num_objects":4617,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15546,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15483","reported_seq":16364,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.472680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:36.472680+0000","last_peered":"2024-04-22T08:34:36.472680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:36.472680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264722,"num_objects":4705,"num_object_clones":0,"num_object_copies":9410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15481,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15328","reported_seq":15860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.978690+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:35.978690+0000","last_peered":"2024-04-22T08:34:35.978690+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:35.978690+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18845696,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15351","reported_seq":23029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.979203+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:35.979203+0000","last_peered":"2024-04-22T08:34:35.979203+0000","last_clean":"2024-04-22T08:34:35.979203+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:35.979203+0000","last_undegraded":"2024-04-22T08:34:35.979203+0000","last_fullsized":"2024-04-22T08:34:35.979203+0000","mapping_epoch":38,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773202,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15349,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15405","reported_seq":16160,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.463406+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:36.463406+0000","last_peered":"2024-04-22T08:34:36.463406+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:36.463406+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19403369,"num_objects":4738,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15404,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15274","reported_seq":15946,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.480273+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:36.480273+0000","last_peered":"2024-04-22T08:34:36.480273+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:36.480273+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18333696,"num_objects":4476,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15380","reported_seq":15982,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.464587+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:36.464587+0000","last_peered":"2024-04-22T08:34:36.464587+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:36.464587+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19080402,"num_objects":4660,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15451","reported_seq":16165,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.393999+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:36.393999+0000","last_peered":"2024-04-22T08:34:36.393999+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:36.393999+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18643364,"num_objects":4555,"num_object_clones":0,"num_object_copies":9110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15448,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":302137197,"num_objects":73776,"num_object_clones":0,"num_object_copies":147552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219928,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13294,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":791756800,"data_stored":791682555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160553,"ondisk_log_size":160553,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117087,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601972,"kb_used_data":269292,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93769868,"statfs":{"total":96636764160,"available":96020344832,"internally_reserved":0,"allocated":275755008,"data_stored":275529081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.255}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920849,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583988,"kb_used_data":223404,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93787852,"statfs":{"total":96636764160,"available":96038760448,"internally_reserved":0,"allocated":228765696,"data_stored":228484105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.44400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313315,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679672,"kb_used_data":283056,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93692168,"statfs":{"total":96636764160,"available":95940780032,"internally_reserved":0,"allocated":289849344,"data_stored":289633951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406122195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.25},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288948224,"data_stored":288944745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":227483648,"data_stored":227420433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275324928,"data_stored":275317377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:38.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'15525", 'reported_seq': 16244, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.463593+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:36.463593+0000', 'last_peered': '2024-04-22T08:34:36.463593+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:36.463593+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'5500", 'ondisk_log_start': "35'5500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19017728, 'num_objects': 4643, 'num_object_clones': 0, 'num_object_copies': 9286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15525, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1175}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15307", 'reported_seq': 15841, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:35.880157+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:35.880157+0000', 'last_peered': '2024-04-22T08:34:35.880157+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:35.880157+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5300", 'ondisk_log_start': "35'5300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18661400, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 9114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12134, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'15514", 'reported_seq': 22229, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.473477+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:36.473477+0000', 'last_peered': '2024-04-22T08:34:36.473477+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:36.473477+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'5500", 'ondisk_log_start': "35'5500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18567168, 'num_objects': 4533, 'num_object_clones': 0, 'num_object_copies': 9066, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 607, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12383, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3275, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3926}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'15606", 'reported_seq': 16062, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.044788+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:36.044788+0000', 'last_peered': '2024-04-22T08:34:36.044788+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:36.044788+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18904274, 'num_objects': 4617, 'num_object_clones': 0, 'num_object_copies': 9234, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15546, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:38.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'15483", 'reported_seq': 16364, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.472680+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:36.472680+0000', 'last_peered': '2024-04-22T08:34:36.472680+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:36.472680+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'5400", 'ondisk_log_start': "35'5400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19264722, 'num_objects': 4705, 'num_object_clones': 0, 'num_object_copies': 9410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15481, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1334}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'15328", 'reported_seq': 15860, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:35.978690+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:35.978690+0000', 'last_peered': '2024-04-22T08:34:35.978690+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:35.978690+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'5300", 'ondisk_log_start': "35'5300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18845696, 'num_objects': 4601, 'num_object_clones': 0, 'num_object_copies': 9202, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11782, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1042}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'15405", 'reported_seq': 16160, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.463406+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:36.463406+0000', 'last_peered': '2024-04-22T08:34:36.463406+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:36.463406+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'5400", 'ondisk_log_start': "35'5400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19403369, 'num_objects': 4738, 'num_object_clones': 0, 'num_object_copies': 9476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15404, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1353}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'15274", 'reported_seq': 15946, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.480273+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:36.480273+0000', 'last_peered': '2024-04-22T08:34:36.480273+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:36.480273+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'5200", 'ondisk_log_start': "35'5200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18333696, 'num_objects': 4476, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11505, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'15380", 'reported_seq': 15982, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.464587+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:36.464587+0000', 'last_peered': '2024-04-22T08:34:36.464587+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:36.464587+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'5300", 'ondisk_log_start': "35'5300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19080402, 'num_objects': 4660, 'num_object_clones': 0, 'num_object_copies': 9320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15378, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:38.926 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'15451", 'reported_seq': 16165, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:36.393999+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:36.393999+0000', 'last_peered': '2024-04-22T08:34:36.393999+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:36.393999+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'5400", 'ondisk_log_start': "35'5400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18643364, 'num_objects': 4555, 'num_object_clones': 0, 'num_object_copies': 9110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15448, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:38.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:39.855 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:39.855 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32313/147552 objects degraded (21.899%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":514,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":73776,"data_bytes":302137197,"bytes_used":1910407168,"bytes_avail":287999885312,"bytes_total":289910292480,"degraded_objects":32313,"degraded_total":147552,"degraded_ratio":0.21899398178269355,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":497},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-22T08:33:26.301392+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:39.917 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:39.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:40.642 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:40.719 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-22T08:34:40.334958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302137197,"num_objects":73776,"num_object_clones":0,"num_object_copies":147552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219928,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13294,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160553,"ondisk_log_size":160553,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1865632,"kb_used_data":775752,"kb_used_omap":12,"kb_used_meta":1089843,"kb_avail":281249888,"statfs":{"total":289910292480,"available":287999885312,"internally_reserved":0,"allocated":794370048,"data_stored":793647137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1115999342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,7,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18849792,"num_objects":-4602,"num_object_clones":0,"num_object_copies":-9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007006"},"pg_stats":[{"pgid":"2.f","version":"39'15525","reported_seq":16244,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.463593+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:36.463593+0000","last_peered":"2024-04-22T08:34:36.463593+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:36.463593+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19017728,"num_objects":4643,"num_object_clones":0,"num_object_copies":9286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15525,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15516","reported_seq":15635,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.995667+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:35.995667+0000","last_peered":"2024-04-22T08:34:35.995667+0000","last_clean":"2024-04-22T08:34:35.995667+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:35.995667+0000","last_undegraded":"2024-04-22T08:34:35.995667+0000","last_fullsized":"2024-04-22T08:34:35.995667+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19316736,"num_objects":4716,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12572,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15307","reported_seq":15841,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.880157+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:35.880157+0000","last_peered":"2024-04-22T08:34:35.880157+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:35.880157+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661400,"num_objects":4557,"num_object_clones":0,"num_object_copies":9114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12134,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15503","reported_seq":15537,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.996261+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:35.996261+0000","last_peered":"2024-04-22T08:34:35.996261+0000","last_clean":"2024-04-22T08:34:35.996261+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:35.996261+0000","last_undegraded":"2024-04-22T08:34:35.996261+0000","last_fullsized":"2024-04-22T08:34:35.996261+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15503,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15345","reported_seq":15487,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:36.343841+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:36.343841+0000","last_peered":"2024-04-22T08:34:36.343841+0000","last_clean":"2024-04-22T08:34:36.343841+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:36.343841+0000","last_undegraded":"2024-04-22T08:34:36.343841+0000","last_fullsized":"2024-04-22T08:34:36.343841+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18735104,"num_objects":4574,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12437,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15514","reported_seq":22229,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.473477+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:36.473477+0000","last_peered":"2024-04-22T08:34:36.473477+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:36.473477+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18567168,"num_objects":4533,"num_object_clones":0,"num_object_copies":9066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3926}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15231","reported_seq":22971,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.558945+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:35.558945+0000","last_peered":"2024-04-22T08:34:35.558945+0000","last_clean":"2024-04-22T08:34:35.558945+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:35.558945+0000","last_undegraded":"2024-04-22T08:34:35.558945+0000","last_fullsized":"2024-04-22T08:34:35.558945+0000","mapping_epoch":36,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15234","reported_seq":15382,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:36.405966+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:36.405966+0000","last_peered":"2024-04-22T08:34:36.405966+0000","last_clean":"2024-04-22T08:34:36.405966+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:36.405966+0000","last_undegraded":"2024-04-22T08:34:36.405966+0000","last_fullsized":"2024-04-22T08:34:36.405966+0000","mapping_epoch":27,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18726912,"num_objects":4572,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15606","reported_seq":16062,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.044788+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:36.044788+0000","last_peered":"2024-04-22T08:34:36.044788+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:36.044788+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904274,"num_objects":4617,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15546,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15483","reported_seq":16364,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.472680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:36.472680+0000","last_peered":"2024-04-22T08:34:36.472680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:36.472680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264722,"num_objects":4705,"num_object_clones":0,"num_object_copies":9410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15481,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15328","reported_seq":15860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:35.978690+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:35.978690+0000","last_peered":"2024-04-22T08:34:35.978690+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:35.978690+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18845696,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15351","reported_seq":23029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:35.979203+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:35.979203+0000","last_peered":"2024-04-22T08:34:35.979203+0000","last_clean":"2024-04-22T08:34:35.979203+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:35.979203+0000","last_undegraded":"2024-04-22T08:34:35.979203+0000","last_fullsized":"2024-04-22T08:34:35.979203+0000","mapping_epoch":38,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773202,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15349,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15405","reported_seq":16160,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.463406+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:36.463406+0000","last_peered":"2024-04-22T08:34:36.463406+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:36.463406+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19403369,"num_objects":4738,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15404,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15274","reported_seq":15946,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.480273+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:36.480273+0000","last_peered":"2024-04-22T08:34:36.480273+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:36.480273+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5200","ondisk_log_start":"35'5200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18333696,"num_objects":4476,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15380","reported_seq":15982,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.464587+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:36.464587+0000","last_peered":"2024-04-22T08:34:36.464587+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:36.464587+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19080402,"num_objects":4660,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15451","reported_seq":16165,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:36.393999+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:36.393999+0000","last_peered":"2024-04-22T08:34:36.393999+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:36.393999+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18643364,"num_objects":4555,"num_object_clones":0,"num_object_copies":9110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15448,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":302137197,"num_objects":73776,"num_object_clones":0,"num_object_copies":147552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219928,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13294,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":791756800,"data_stored":791682555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160553,"ondisk_log_size":160553,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117087,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601972,"kb_used_data":269292,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93769868,"statfs":{"total":96636764160,"available":96020344832,"internally_reserved":0,"allocated":275755008,"data_stored":275529081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.255}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.26600000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920849,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583988,"kb_used_data":223404,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93787852,"statfs":{"total":96636764160,"available":96038760448,"internally_reserved":0,"allocated":228765696,"data_stored":228484105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":369225420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.44400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313315,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679672,"kb_used_data":283056,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93692168,"statfs":{"total":96636764160,"available":95940780032,"internally_reserved":0,"allocated":289849344,"data_stored":289633951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":406122195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.25},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288948224,"data_stored":288944745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":227483648,"data_stored":227420433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275324928,"data_stored":275317377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:43.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:45.041 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:45.106 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-22T08:34:44.335975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293261165,"num_objects":71609,"num_object_clones":0,"num_object_copies":143218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222095,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13418,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1866024,"kb_used_data":758092,"kb_used_omap":12,"kb_used_meta":1107891,"kb_avail":281249496,"statfs":{"total":289910292480,"available":287999483904,"internally_reserved":0,"allocated":776286208,"data_stored":775563297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1134480494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,3,6,4,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18599936,"num_objects":-4541,"num_object_clones":0,"num_object_copies":-9082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006736"},"pg_stats":[{"pgid":"2.f","version":"39'15651","reported_seq":16370,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.408815+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:41.408815+0000","last_peered":"2024-04-22T08:34:41.408815+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:41.408815+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18501632,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15651,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15657","reported_seq":15776,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.828089+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:40.828089+0000","last_peered":"2024-04-22T08:34:40.828089+0000","last_clean":"2024-04-22T08:34:40.828089+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:40.828089+0000","last_undegraded":"2024-04-22T08:34:40.828089+0000","last_fullsized":"2024-04-22T08:34:40.828089+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739200,"num_objects":4575,"num_object_clones":0,"num_object_copies":9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12713,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15432","reported_seq":15966,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.792537+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:40.792537+0000","last_peered":"2024-04-22T08:34:40.792537+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:40.792537+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18149400,"num_objects":4432,"num_object_clones":0,"num_object_copies":8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12259,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15641","reported_seq":15675,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.836954+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:40.836954+0000","last_peered":"2024-04-22T08:34:40.836954+0000","last_clean":"2024-04-22T08:34:40.836954+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:40.836954+0000","last_undegraded":"2024-04-22T08:34:40.836954+0000","last_fullsized":"2024-04-22T08:34:40.836954+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18321408,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15641,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15483","reported_seq":15625,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:41.433355+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:41.433355+0000","last_peered":"2024-04-22T08:34:41.433355+0000","last_clean":"2024-04-22T08:34:41.433355+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:41.433355+0000","last_undegraded":"2024-04-22T08:34:41.433355+0000","last_fullsized":"2024-04-22T08:34:41.433355+0000","mapping_epoch":27,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169856,"num_objects":4436,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15644","reported_seq":22607,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.478441+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:41.478441+0000","last_peered":"2024-04-22T08:34:41.478441+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:41.478441+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18034688,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3399,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3920}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15365","reported_seq":23105,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.683308+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:40.683308+0000","last_peered":"2024-04-22T08:34:40.683308+0000","last_clean":"2024-04-22T08:34:40.683308+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:40.683308+0000","last_undegraded":"2024-04-22T08:34:40.683308+0000","last_fullsized":"2024-04-22T08:34:40.683308+0000","mapping_epoch":36,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15366","reported_seq":15514,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:41.471711+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:41.471711+0000","last_peered":"2024-04-22T08:34:41.471711+0000","last_clean":"2024-04-22T08:34:41.471711+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:41.471711+0000","last_undegraded":"2024-04-22T08:34:41.471711+0000","last_fullsized":"2024-04-22T08:34:41.471711+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18186240,"num_objects":4440,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12014,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15732","reported_seq":16189,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.859959+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:40.859959+0000","last_peered":"2024-04-22T08:34:40.859959+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:40.859959+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388178,"num_objects":4491,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15672,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15641","reported_seq":16522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.475636+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:41.475636+0000","last_peered":"2024-04-22T08:34:41.475636+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:41.475636+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617554,"num_objects":4547,"num_object_clones":0,"num_object_copies":9094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15639,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15477","reported_seq":16009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.852230+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:40.852230+0000","last_peered":"2024-04-22T08:34:40.852230+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:40.852230+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18235392,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15492","reported_seq":23170,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.831062+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:40.831062+0000","last_peered":"2024-04-22T08:34:40.831062+0000","last_clean":"2024-04-22T08:34:40.831062+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:40.831062+0000","last_undegraded":"2024-04-22T08:34:40.831062+0000","last_fullsized":"2024-04-22T08:34:40.831062+0000","mapping_epoch":38,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18195666,"num_objects":4444,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15490,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15548","reported_seq":16300,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.442139+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:41.442139+0000","last_peered":"2024-04-22T08:34:41.442139+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:41.442139+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18817641,"num_objects":4595,"num_object_clones":0,"num_object_copies":9190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15395","reported_seq":16068,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.460141+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:41.460141+0000","last_peered":"2024-04-22T08:34:41.460141+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:41.460141+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17838080,"num_objects":4355,"num_object_clones":0,"num_object_copies":8710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15520","reported_seq":16122,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.587793+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:41.587793+0000","last_peered":"2024-04-22T08:34:41.587793+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:41.587793+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18506962,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15518,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15576","reported_seq":16290,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.420909+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:41.420909+0000","last_peered":"2024-04-22T08:34:41.420909+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:41.420909+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18131364,"num_objects":4430,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15573,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293261165,"num_objects":71609,"num_object_clones":0,"num_object_copies":143218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222095,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13418,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":773754880,"data_stored":773680635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117088,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585272,"kb_used_data":264880,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93786568,"statfs":{"total":96636764160,"available":96037445632,"internally_reserved":0,"allocated":271237120,"data_stored":271011193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":1.915},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.2389999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":2.1269999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599332,"kb_used_data":216604,"kb_used_omap":4,"kb_used_meta":382715,"kb_avail":93772508,"statfs":{"total":96636764160,"available":96023048192,"internally_reserved":0,"allocated":221802496,"data_stored":221520905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":391900876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.252},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":7.9820000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":8.0470000000000006}]}]},{"osd":0,"up_from":21,"seq":90194313316,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681420,"kb_used_data":276608,"kb_used_omap":4,"kb_used_meta":404795,"kb_avail":93690420,"statfs":{"total":96636764160,"available":95938990080,"internally_reserved":0,"allocated":283246592,"data_stored":283031199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":414510803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,4,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.252}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.378},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282292224,"data_stored":282288745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220585984,"data_stored":220522769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270876672,"data_stored":270869121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:45.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:45.880 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:45.882 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:45.955 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-22T08:34:44.335975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293261165,"num_objects":71609,"num_object_clones":0,"num_object_copies":143218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222095,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13418,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1866024,"kb_used_data":758092,"kb_used_omap":12,"kb_used_meta":1107891,"kb_avail":281249496,"statfs":{"total":289910292480,"available":287999483904,"internally_reserved":0,"allocated":776286208,"data_stored":775563297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1134480494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,3,6,4,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18599936,"num_objects":-4541,"num_object_clones":0,"num_object_copies":-9082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006736"},"pg_stats":[{"pgid":"2.f","version":"39'15651","reported_seq":16370,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.408815+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:41.408815+0000","last_peered":"2024-04-22T08:34:41.408815+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:41.408815+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18501632,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15651,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15657","reported_seq":15776,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.828089+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:40.828089+0000","last_peered":"2024-04-22T08:34:40.828089+0000","last_clean":"2024-04-22T08:34:40.828089+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:40.828089+0000","last_undegraded":"2024-04-22T08:34:40.828089+0000","last_fullsized":"2024-04-22T08:34:40.828089+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739200,"num_objects":4575,"num_object_clones":0,"num_object_copies":9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12713,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15432","reported_seq":15966,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.792537+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:40.792537+0000","last_peered":"2024-04-22T08:34:40.792537+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:40.792537+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18149400,"num_objects":4432,"num_object_clones":0,"num_object_copies":8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12259,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15641","reported_seq":15675,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.836954+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:40.836954+0000","last_peered":"2024-04-22T08:34:40.836954+0000","last_clean":"2024-04-22T08:34:40.836954+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:40.836954+0000","last_undegraded":"2024-04-22T08:34:40.836954+0000","last_fullsized":"2024-04-22T08:34:40.836954+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18321408,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15641,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15483","reported_seq":15625,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:41.433355+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:41.433355+0000","last_peered":"2024-04-22T08:34:41.433355+0000","last_clean":"2024-04-22T08:34:41.433355+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:41.433355+0000","last_undegraded":"2024-04-22T08:34:41.433355+0000","last_fullsized":"2024-04-22T08:34:41.433355+0000","mapping_epoch":27,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169856,"num_objects":4436,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15644","reported_seq":22607,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.478441+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:41.478441+0000","last_peered":"2024-04-22T08:34:41.478441+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:41.478441+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18034688,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3399,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3920}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15365","reported_seq":23105,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.683308+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:40.683308+0000","last_peered":"2024-04-22T08:34:40.683308+0000","last_clean":"2024-04-22T08:34:40.683308+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:40.683308+0000","last_undegraded":"2024-04-22T08:34:40.683308+0000","last_fullsized":"2024-04-22T08:34:40.683308+0000","mapping_epoch":36,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15366","reported_seq":15514,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:41.471711+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:41.471711+0000","last_peered":"2024-04-22T08:34:41.471711+0000","last_clean":"2024-04-22T08:34:41.471711+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:41.471711+0000","last_undegraded":"2024-04-22T08:34:41.471711+0000","last_fullsized":"2024-04-22T08:34:41.471711+0000","mapping_epoch":27,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18186240,"num_objects":4440,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12014,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15732","reported_seq":16189,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.859959+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:40.859959+0000","last_peered":"2024-04-22T08:34:40.859959+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:40.859959+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388178,"num_objects":4491,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15672,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15641","reported_seq":16522,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.475636+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:41.475636+0000","last_peered":"2024-04-22T08:34:41.475636+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:41.475636+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617554,"num_objects":4547,"num_object_clones":0,"num_object_copies":9094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15639,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15477","reported_seq":16009,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:40.852230+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:40.852230+0000","last_peered":"2024-04-22T08:34:40.852230+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:40.852230+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18235392,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15492","reported_seq":23170,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:40.831062+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:40.831062+0000","last_peered":"2024-04-22T08:34:40.831062+0000","last_clean":"2024-04-22T08:34:40.831062+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:40.831062+0000","last_undegraded":"2024-04-22T08:34:40.831062+0000","last_fullsized":"2024-04-22T08:34:40.831062+0000","mapping_epoch":38,"log_start":"35'5400","ondisk_log_start":"35'5400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18195666,"num_objects":4444,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15490,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15548","reported_seq":16300,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.442139+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:41.442139+0000","last_peered":"2024-04-22T08:34:41.442139+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:41.442139+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18817641,"num_objects":4595,"num_object_clones":0,"num_object_copies":9190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15395","reported_seq":16068,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.460141+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:41.460141+0000","last_peered":"2024-04-22T08:34:41.460141+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:41.460141+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5300","ondisk_log_start":"35'5300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17838080,"num_objects":4355,"num_object_clones":0,"num_object_copies":8710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15520","reported_seq":16122,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.587793+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:41.587793+0000","last_peered":"2024-04-22T08:34:41.587793+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:41.587793+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18506962,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15518,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15576","reported_seq":16290,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:41.420909+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:41.420909+0000","last_peered":"2024-04-22T08:34:41.420909+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:41.420909+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18131364,"num_objects":4430,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15573,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293261165,"num_objects":71609,"num_object_clones":0,"num_object_copies":143218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222095,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13418,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":773754880,"data_stored":773680635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117088,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585272,"kb_used_data":264880,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93786568,"statfs":{"total":96636764160,"available":96037445632,"internally_reserved":0,"allocated":271237120,"data_stored":271011193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":1.915},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.2389999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":2.1269999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599332,"kb_used_data":216604,"kb_used_omap":4,"kb_used_meta":382715,"kb_avail":93772508,"statfs":{"total":96636764160,"available":96023048192,"internally_reserved":0,"allocated":221802496,"data_stored":221520905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":391900876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.252},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":7.9820000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":8.0470000000000006}]}]},{"osd":0,"up_from":21,"seq":90194313316,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681420,"kb_used_data":276608,"kb_used_omap":4,"kb_used_meta":404795,"kb_avail":93690420,"statfs":{"total":96636764160,"available":95938990080,"internally_reserved":0,"allocated":283246592,"data_stored":283031199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":414510803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,4,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.252}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.378},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282292224,"data_stored":282288745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220585984,"data_stored":220522769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270876672,"data_stored":270869121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:45.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:45.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'15651", 'reported_seq': 16370, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.408815+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:41.408815+0000', 'last_peered': '2024-04-22T08:34:41.408815+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:41.408815+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18501632, 'num_objects': 4517, 'num_object_clones': 0, 'num_object_copies': 9034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15651, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1049}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:45.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15432", 'reported_seq': 15966, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:40.792537+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:40.792537+0000', 'last_peered': '2024-04-22T08:34:40.792537+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:40.792537+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5400", 'ondisk_log_start': "35'5400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18149400, 'num_objects': 4432, 'num_object_clones': 0, 'num_object_copies': 8864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4432, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12259, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'15644", 'reported_seq': 22607, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.478441+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:41.478441+0000', 'last_peered': '2024-04-22T08:34:41.478441+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:41.478441+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18034688, 'num_objects': 4403, 'num_object_clones': 0, 'num_object_copies': 8806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 483, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12513, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3399, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'15732", 'reported_seq': 16189, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:40.859959+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:40.859959+0000', 'last_peered': '2024-04-22T08:34:40.859959+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:40.859959+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'5700", 'ondisk_log_start': "35'5700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18388178, 'num_objects': 4491, 'num_object_clones': 0, 'num_object_copies': 8982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15672, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'15641", 'reported_seq': 16522, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.475636+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:41.475636+0000', 'last_peered': '2024-04-22T08:34:41.475636+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:41.475636+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18617554, 'num_objects': 4547, 'num_object_clones': 0, 'num_object_copies': 9094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15639, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1176}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'15477", 'reported_seq': 16009, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:40.852230+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:40.852230+0000', 'last_peered': '2024-04-22T08:34:40.852230+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:40.852230+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'5400", 'ondisk_log_start': "35'5400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18235392, 'num_objects': 4452, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11931, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 893}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:45.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'15548", 'reported_seq': 16300, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.442139+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:41.442139+0000', 'last_peered': '2024-04-22T08:34:41.442139+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:41.442139+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'5500", 'ondisk_log_start': "35'5500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18817641, 'num_objects': 4595, 'num_object_clones': 0, 'num_object_copies': 9190, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15547, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'15395", 'reported_seq': 16068, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.460141+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:41.460141+0000', 'last_peered': '2024-04-22T08:34:41.460141+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:41.460141+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'5300", 'ondisk_log_start': "35'5300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17838080, 'num_objects': 4355, 'num_object_clones': 0, 'num_object_copies': 8710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11626, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'15520", 'reported_seq': 16122, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.587793+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:41.587793+0000', 'last_peered': '2024-04-22T08:34:41.587793+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:41.587793+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'5500", 'ondisk_log_start': "35'5500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18506962, 'num_objects': 4520, 'num_object_clones': 0, 'num_object_copies': 9040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15518, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:45.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'15576", 'reported_seq': 16290, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:41.420909+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:41.420909+0000', 'last_peered': '2024-04-22T08:34:41.420909+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:41.420909+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'5500", 'ondisk_log_start': "35'5500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18131364, 'num_objects': 4430, 'num_object_clones': 0, 'num_object_copies': 8860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15573, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:45.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:47.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:47.086 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32189/141136 objects degraded (22.807%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":70568,"data_bytes":288997229,"bytes_used":1913520128,"bytes_avail":287996772352,"bytes_total":289910292480,"degraded_objects":32189,"degraded_total":141136,"degraded_ratio":0.22807079696179572,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":465},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:47.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:47.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:48.507 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:48.509 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:48.567 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-22T08:34:48.336894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283553645,"num_objects":69239,"num_object_clones":0,"num_object_copies":138478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224465,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13542,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1849328,"kb_used_data":739736,"kb_used_omap":12,"kb_used_meta":1109555,"kb_avail":281266192,"statfs":{"total":289910292480,"available":288016580608,"internally_reserved":0,"allocated":757489664,"data_stored":756766753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1136184430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,6,3,7,7,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24125440,"num_objects":-5890,"num_object_clones":0,"num_object_copies":-11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5890,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002942"},"pg_stats":[{"pgid":"2.f","version":"39'15818","reported_seq":16537,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.597676+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:46.597676+0000","last_peered":"2024-04-22T08:34:46.597676+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:46.597676+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17817600,"num_objects":4350,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15818,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":882}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15819","reported_seq":15938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.000447+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:46.000447+0000","last_peered":"2024-04-22T08:34:46.000447+0000","last_clean":"2024-04-22T08:34:46.000447+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:46.000447+0000","last_undegraded":"2024-04-22T08:34:46.000447+0000","last_fullsized":"2024-04-22T08:34:46.000447+0000","mapping_epoch":27,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18075648,"num_objects":4413,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12875,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15571","reported_seq":16105,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.020383+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:46.020383+0000","last_peered":"2024-04-22T08:34:46.020383+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:46.020383+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17580056,"num_objects":4293,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12398,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15789","reported_seq":15823,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:45.973962+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:45.973962+0000","last_peered":"2024-04-22T08:34:45.973962+0000","last_clean":"2024-04-22T08:34:45.973962+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:45.973962+0000","last_undegraded":"2024-04-22T08:34:45.973962+0000","last_fullsized":"2024-04-22T08:34:45.973962+0000","mapping_epoch":27,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15625","reported_seq":15767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.595122+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:46.595122+0000","last_peered":"2024-04-22T08:34:46.595122+0000","last_clean":"2024-04-22T08:34:46.595122+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:46.595122+0000","last_undegraded":"2024-04-22T08:34:46.595122+0000","last_fullsized":"2024-04-22T08:34:46.595122+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17588224,"num_objects":4294,"num_object_clones":0,"num_object_copies":8588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12717,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15793","reported_seq":23004,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.605541+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:46.605541+0000","last_peered":"2024-04-22T08:34:46.605541+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:46.605541+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17424384,"num_objects":4254,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12662,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3523,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15511","reported_seq":23251,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:45.668921+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:45.668921+0000","last_peered":"2024-04-22T08:34:45.668921+0000","last_clean":"2024-04-22T08:34:45.668921+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:45.668921+0000","last_undegraded":"2024-04-22T08:34:45.668921+0000","last_fullsized":"2024-04-22T08:34:45.668921+0000","mapping_epoch":36,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17829888,"num_objects":4353,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15504","reported_seq":15652,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.596872+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:46.596872+0000","last_peered":"2024-04-22T08:34:46.596872+0000","last_clean":"2024-04-22T08:34:46.596872+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:46.596872+0000","last_undegraded":"2024-04-22T08:34:46.596872+0000","last_fullsized":"2024-04-22T08:34:46.596872+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":4302,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12152,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15870","reported_seq":16327,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.049398+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:46.049398+0000","last_peered":"2024-04-22T08:34:46.049398+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:46.049398+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17822930,"num_objects":4353,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15810,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":630}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15793","reported_seq":16674,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.612034+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:46.612034+0000","last_peered":"2024-04-22T08:34:46.612034+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:46.612034+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994962,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15791,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15618","reported_seq":16150,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:45.949700+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:45.949700+0000","last_peered":"2024-04-22T08:34:45.949700+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:45.949700+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17657856,"num_objects":4311,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":752}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15659","reported_seq":23337,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.039153+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:46.039153+0000","last_peered":"2024-04-22T08:34:46.039153+0000","last_clean":"2024-04-22T08:34:46.039153+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:46.039153+0000","last_undegraded":"2024-04-22T08:34:46.039153+0000","last_fullsized":"2024-04-22T08:34:46.039153+0000","mapping_epoch":38,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17511634,"num_objects":4277,"num_object_clones":0,"num_object_copies":8554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15657,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15669","reported_seq":16424,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.585397+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:46.585397+0000","last_peered":"2024-04-22T08:34:46.585397+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:46.585397+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18322025,"num_objects":4474,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15668,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15538","reported_seq":16211,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.549686+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:46.549686+0000","last_peered":"2024-04-22T08:34:46.549686+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:46.549686+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15682","reported_seq":16284,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.578208+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:46.578208+0000","last_peered":"2024-04-22T08:34:46.578208+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:46.578208+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17843410,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15680,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15731","reported_seq":16445,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.568737+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:46.568737+0000","last_peered":"2024-04-22T08:34:46.568737+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:46.568737+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17496484,"num_objects":4275,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15728,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":283553645,"num_objects":69239,"num_object_clones":0,"num_object_copies":138478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224465,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13542,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":754630656,"data_stored":754556411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117089,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584784,"kb_used_data":260296,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93787056,"statfs":{"total":96636764160,"available":96037945344,"internally_reserved":0,"allocated":266543104,"data_stored":266317177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,0,3,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.45700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920851,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602468,"kb_used_data":209884,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93769372,"statfs":{"total":96636764160,"available":96019836928,"internally_reserved":0,"allocated":214921216,"data_stored":214639625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":401993420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.242}]}]},{"osd":0,"up_from":21,"seq":90194313317,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662076,"kb_used_data":269556,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93709764,"statfs":{"total":96636764160,"available":95958798336,"internally_reserved":0,"allocated":276025344,"data_stored":275809951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,0,4,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274948096,"data_stored":274944617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213606400,"data_stored":213543185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266076160,"data_stored":266068609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:51.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:52.261 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:52.264 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:52.339 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-04-22T08:34:50.337378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283553645,"num_objects":69239,"num_object_clones":0,"num_object_copies":138478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224465,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13542,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1849328,"kb_used_data":739736,"kb_used_omap":12,"kb_used_meta":1109555,"kb_avail":281266192,"statfs":{"total":289910292480,"available":288016580608,"internally_reserved":0,"allocated":757489664,"data_stored":756766753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1136184430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,6,3,7,7,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18583552,"num_objects":-4537,"num_object_clones":0,"num_object_copies":-9074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4537,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002913"},"pg_stats":[{"pgid":"2.f","version":"39'15818","reported_seq":16537,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.597676+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:46.597676+0000","last_peered":"2024-04-22T08:34:46.597676+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:46.597676+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17817600,"num_objects":4350,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15818,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":882}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15819","reported_seq":15938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.000447+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:46.000447+0000","last_peered":"2024-04-22T08:34:46.000447+0000","last_clean":"2024-04-22T08:34:46.000447+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:46.000447+0000","last_undegraded":"2024-04-22T08:34:46.000447+0000","last_fullsized":"2024-04-22T08:34:46.000447+0000","mapping_epoch":27,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18075648,"num_objects":4413,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12875,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15571","reported_seq":16105,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.020383+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:46.020383+0000","last_peered":"2024-04-22T08:34:46.020383+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:46.020383+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17580056,"num_objects":4293,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12398,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":741}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15789","reported_seq":15823,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:45.973962+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:45.973962+0000","last_peered":"2024-04-22T08:34:45.973962+0000","last_clean":"2024-04-22T08:34:45.973962+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:45.973962+0000","last_undegraded":"2024-04-22T08:34:45.973962+0000","last_fullsized":"2024-04-22T08:34:45.973962+0000","mapping_epoch":27,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15625","reported_seq":15767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.595122+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:46.595122+0000","last_peered":"2024-04-22T08:34:46.595122+0000","last_clean":"2024-04-22T08:34:46.595122+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:46.595122+0000","last_undegraded":"2024-04-22T08:34:46.595122+0000","last_fullsized":"2024-04-22T08:34:46.595122+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17588224,"num_objects":4294,"num_object_clones":0,"num_object_copies":8588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12717,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15793","reported_seq":23004,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.605541+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:46.605541+0000","last_peered":"2024-04-22T08:34:46.605541+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:46.605541+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17424384,"num_objects":4254,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12662,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3523,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15511","reported_seq":23251,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:45.668921+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:45.668921+0000","last_peered":"2024-04-22T08:34:45.668921+0000","last_clean":"2024-04-22T08:34:45.668921+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:45.668921+0000","last_undegraded":"2024-04-22T08:34:45.668921+0000","last_fullsized":"2024-04-22T08:34:45.668921+0000","mapping_epoch":36,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17829888,"num_objects":4353,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15504","reported_seq":15652,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.596872+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:46.596872+0000","last_peered":"2024-04-22T08:34:46.596872+0000","last_clean":"2024-04-22T08:34:46.596872+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:46.596872+0000","last_undegraded":"2024-04-22T08:34:46.596872+0000","last_fullsized":"2024-04-22T08:34:46.596872+0000","mapping_epoch":27,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":4302,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12152,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'15870","reported_seq":16327,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.049398+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:46.049398+0000","last_peered":"2024-04-22T08:34:46.049398+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:46.049398+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17822930,"num_objects":4353,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15810,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":630}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15793","reported_seq":16674,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.612034+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:46.612034+0000","last_peered":"2024-04-22T08:34:46.612034+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:46.612034+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17994962,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15791,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15618","reported_seq":16150,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:45.949700+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:45.949700+0000","last_peered":"2024-04-22T08:34:45.949700+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:45.949700+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17657856,"num_objects":4311,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":752}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15659","reported_seq":23337,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:46.039153+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:46.039153+0000","last_peered":"2024-04-22T08:34:46.039153+0000","last_clean":"2024-04-22T08:34:46.039153+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:46.039153+0000","last_undegraded":"2024-04-22T08:34:46.039153+0000","last_fullsized":"2024-04-22T08:34:46.039153+0000","mapping_epoch":38,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17511634,"num_objects":4277,"num_object_clones":0,"num_object_copies":8554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15657,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15669","reported_seq":16424,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.585397+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:46.585397+0000","last_peered":"2024-04-22T08:34:46.585397+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:46.585397+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18322025,"num_objects":4474,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15668,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15538","reported_seq":16211,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.549686+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:46.549686+0000","last_peered":"2024-04-22T08:34:46.549686+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:46.549686+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5500","ondisk_log_start":"35'5500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15682","reported_seq":16284,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.578208+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:46.578208+0000","last_peered":"2024-04-22T08:34:46.578208+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:46.578208+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17843410,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15680,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15731","reported_seq":16445,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:46.568737+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:46.568737+0000","last_peered":"2024-04-22T08:34:46.568737+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:46.568737+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17496484,"num_objects":4275,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15728,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":283553645,"num_objects":69239,"num_object_clones":0,"num_object_copies":138478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224465,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13542,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":754630656,"data_stored":754556411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117089,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584784,"kb_used_data":260296,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93787056,"statfs":{"total":96636764160,"available":96037945344,"internally_reserved":0,"allocated":266543104,"data_stored":266317177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332263119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,0,3,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.45700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920851,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602468,"kb_used_data":209884,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93769372,"statfs":{"total":96636764160,"available":96019836928,"internally_reserved":0,"allocated":214921216,"data_stored":214639625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":401993420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.242}]}]},{"osd":0,"up_from":21,"seq":90194313317,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662076,"kb_used_data":269556,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93709764,"statfs":{"total":96636764160,"available":95958798336,"internally_reserved":0,"allocated":276025344,"data_stored":275809951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":401927891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,0,4,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274948096,"data_stored":274944617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213606400,"data_stored":213543185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266076160,"data_stored":266068609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:52.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:53.139 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-22T08:34:52.337759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":273756013,"num_objects":66847,"num_object_clones":0,"num_object_copies":133694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226857,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1868824,"kb_used_data":720684,"kb_used_omap":12,"kb_used_meta":1148083,"kb_avail":281246696,"statfs":{"total":289910292480,"available":287996616704,"internally_reserved":0,"allocated":737980416,"data_stored":737257505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1175637102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,8,6,20],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28381184,"num_objects":-6929,"num_object_clones":0,"num_object_copies":-13858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002800"},"pg_stats":[{"pgid":"2.f","version":"39'15993","reported_seq":16709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.604030+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:51.604030+0000","last_peered":"2024-04-22T08:34:51.604030+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:51.604030+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17100800,"num_objects":4175,"num_object_clones":0,"num_object_copies":8350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15993,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":707}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15951","reported_seq":16069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.092390+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:51.092390+0000","last_peered":"2024-04-22T08:34:51.092390+0000","last_clean":"2024-04-22T08:34:51.092390+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:51.092390+0000","last_undegraded":"2024-04-22T08:34:51.092390+0000","last_fullsized":"2024-04-22T08:34:51.092390+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17534976,"num_objects":4281,"num_object_clones":0,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13007,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15693","reported_seq":16227,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.077842+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:51.077842+0000","last_peered":"2024-04-22T08:34:51.077842+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:51.077842+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17080344,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12520,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15928","reported_seq":15962,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:50.996966+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:50.996966+0000","last_peered":"2024-04-22T08:34:50.996966+0000","last_clean":"2024-04-22T08:34:50.996966+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:50.996966+0000","last_undegraded":"2024-04-22T08:34:50.996966+0000","last_fullsized":"2024-04-22T08:34:50.996966+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17145856,"num_objects":4186,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15777","reported_seq":15919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.579841+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:51.579841+0000","last_peered":"2024-04-22T08:34:51.579841+0000","last_clean":"2024-04-22T08:34:51.579841+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:51.579841+0000","last_undegraded":"2024-04-22T08:34:51.579841+0000","last_fullsized":"2024-04-22T08:34:51.579841+0000","mapping_epoch":27,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16965632,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12869,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15951","reported_seq":23432,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.654245+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:51.654245+0000","last_peered":"2024-04-22T08:34:51.654245+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:51.654245+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4096,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12820,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3658,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15670","reported_seq":23410,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:50.699818+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:50.699818+0000","last_peered":"2024-04-22T08:34:50.699818+0000","last_clean":"2024-04-22T08:34:50.699818+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:50.699818+0000","last_undegraded":"2024-04-22T08:34:50.699818+0000","last_fullsized":"2024-04-22T08:34:50.699818+0000","mapping_epoch":36,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15670","reported_seq":15818,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.626940+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:51.626940+0000","last_peered":"2024-04-22T08:34:51.626940+0000","last_clean":"2024-04-22T08:34:51.626940+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:51.626940+0000","last_undegraded":"2024-04-22T08:34:51.626940+0000","last_fullsized":"2024-04-22T08:34:51.626940+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16941056,"num_objects":4136,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16001","reported_seq":16458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.095683+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:51.095683+0000","last_peered":"2024-04-22T08:34:51.095683+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:51.095683+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17286354,"num_objects":4222,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15935","reported_seq":16816,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.465849+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:51.465849+0000","last_peered":"2024-04-22T08:34:51.465849+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:51.465849+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413330,"num_objects":4253,"num_object_clones":0,"num_object_copies":8506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15933,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":882}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15758","reported_seq":16290,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.053442+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:51.053442+0000","last_peered":"2024-04-22T08:34:51.053442+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:51.053442+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17084416,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12212,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15814","reported_seq":23492,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.076334+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:51.076334+0000","last_peered":"2024-04-22T08:34:51.076334+0000","last_clean":"2024-04-22T08:34:51.076334+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:51.076334+0000","last_undegraded":"2024-04-22T08:34:51.076334+0000","last_fullsized":"2024-04-22T08:34:51.076334+0000","mapping_epoch":38,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16876754,"num_objects":4122,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15812,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15856","reported_seq":16608,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.601736+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:51.601736+0000","last_peered":"2024-04-22T08:34:51.601736+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:51.601736+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17556073,"num_objects":4287,"num_object_clones":0,"num_object_copies":8574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15855,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15677","reported_seq":16350,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.515481+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:51.515481+0000","last_peered":"2024-04-22T08:34:51.515481+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:51.515481+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16683008,"num_objects":4073,"num_object_clones":0,"num_object_copies":8146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15828","reported_seq":16430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.658118+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:51.658118+0000","last_peered":"2024-04-22T08:34:51.658118+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:51.658118+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17245394,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15826,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15880","reported_seq":16592,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.617021+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:51.617021+0000","last_peered":"2024-04-22T08:34:51.617021+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:51.617021+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16886180,"num_objects":4126,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15877,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":273756013,"num_objects":66847,"num_object_clones":0,"num_object_copies":133694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226857,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735064064,"data_stored":734989819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117090,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584360,"kb_used_data":255768,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93787480,"statfs":{"total":96636764160,"available":96038379520,"internally_reserved":0,"allocated":261906432,"data_stored":261680505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.246},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.28000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621464,"kb_used_data":202640,"kb_used_omap":4,"kb_used_meta":418811,"kb_avail":93750376,"statfs":{"total":96636764160,"available":96000385024,"internally_reserved":0,"allocated":207503360,"data_stored":207221769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":428863180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.242}]}]},{"osd":0,"up_from":21,"seq":90194313318,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663000,"kb_used_data":262276,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93708840,"statfs":{"total":96636764160,"available":95957852160,"internally_reserved":0,"allocated":268570624,"data_stored":268355231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":7.1790000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":6.7729999999999997},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267436032,"data_stored":267432553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206188544,"data_stored":206125329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261439488,"data_stored":261431937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'15993", 'reported_seq': 16709, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.604030+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:51.604030+0000', 'last_peered': '2024-04-22T08:34:51.604030+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:51.604030+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17100800, 'num_objects': 4175, 'num_object_clones': 0, 'num_object_copies': 8350, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15993, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 707}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15693", 'reported_seq': 16227, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.077842+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:51.077842+0000', 'last_peered': '2024-04-22T08:34:51.077842+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:51.077842+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17080344, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12520, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 619}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'15951", 'reported_seq': 23432, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.654245+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:51.654245+0000', 'last_peered': '2024-04-22T08:34:51.654245+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:51.654245+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16777216, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 8192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12820, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3658, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:53.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'16001", 'reported_seq': 16458, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.095683+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:51.095683+0000', 'last_peered': '2024-04-22T08:34:51.095683+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:51.095683+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17286354, 'num_objects': 4222, 'num_object_clones': 0, 'num_object_copies': 8444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15941, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'15935", 'reported_seq': 16816, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.465849+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:51.465849+0000', 'last_peered': '2024-04-22T08:34:51.465849+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:51.465849+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17413330, 'num_objects': 4253, 'num_object_clones': 0, 'num_object_copies': 8506, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15933, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 882}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'15758", 'reported_seq': 16290, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.053442+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:51.053442+0000', 'last_peered': '2024-04-22T08:34:51.053442+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:51.053442+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'5700", 'ondisk_log_start': "35'5700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17084416, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12212, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 612}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'15856", 'reported_seq': 16608, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.601736+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:51.601736+0000', 'last_peered': '2024-04-22T08:34:51.601736+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:51.601736+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'5800", 'ondisk_log_start': "35'5800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17556073, 'num_objects': 4287, 'num_object_clones': 0, 'num_object_copies': 8574, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15855, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'15677", 'reported_seq': 16350, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.515481+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:51.515481+0000', 'last_peered': '2024-04-22T08:34:51.515481+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:51.515481+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'5600", 'ondisk_log_start': "35'5600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16683008, 'num_objects': 4073, 'num_object_clones': 0, 'num_object_copies': 8146, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11908, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'15828", 'reported_seq': 16430, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.658118+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:51.658118+0000', 'last_peered': '2024-04-22T08:34:51.658118+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:51.658118+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'5800", 'ondisk_log_start': "35'5800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17245394, 'num_objects': 4212, 'num_object_clones': 0, 'num_object_copies': 8424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15826, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:53.143 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'15880", 'reported_seq': 16592, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:51.617021+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:51.617021+0000', 'last_peered': '2024-04-22T08:34:51.617021+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:51.617021+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'5800", 'ondisk_log_start': "35'5800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16886180, 'num_objects': 4126, 'num_object_clones': 0, 'num_object_copies': 8252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15877, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:53.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:34:54.071 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:54.071 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31930/133694 objects degraded (23.883%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":529,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":66847,"data_bytes":273756013,"bytes_used":1913675776,"bytes_avail":287996616704,"bytes_total":289910292480,"degraded_objects":31930,"degraded_total":133694,"degraded_ratio":0.23882896764252695,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":383,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":577},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:34:54.136 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:34:54.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:54.948 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:54.950 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:55.006 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-22T08:34:54.338323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":273756013,"num_objects":66847,"num_object_clones":0,"num_object_copies":133694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226857,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1868824,"kb_used_data":720684,"kb_used_omap":12,"kb_used_meta":1148083,"kb_avail":281246696,"statfs":{"total":289910292480,"available":287996616704,"internally_reserved":0,"allocated":737980416,"data_stored":737257505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1175637102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,8,6,20],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19505152,"num_objects":-4762,"num_object_clones":0,"num_object_copies":-9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002905"},"pg_stats":[{"pgid":"2.f","version":"39'15993","reported_seq":16709,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.604030+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:51.604030+0000","last_peered":"2024-04-22T08:34:51.604030+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:51.604030+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17100800,"num_objects":4175,"num_object_clones":0,"num_object_copies":8350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15993,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":707}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'15951","reported_seq":16069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.092390+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:51.092390+0000","last_peered":"2024-04-22T08:34:51.092390+0000","last_clean":"2024-04-22T08:34:51.092390+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:51.092390+0000","last_undegraded":"2024-04-22T08:34:51.092390+0000","last_fullsized":"2024-04-22T08:34:51.092390+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17534976,"num_objects":4281,"num_object_clones":0,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13007,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15693","reported_seq":16227,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.077842+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:51.077842+0000","last_peered":"2024-04-22T08:34:51.077842+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:51.077842+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17080344,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12520,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'15928","reported_seq":15962,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:50.996966+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:50.996966+0000","last_peered":"2024-04-22T08:34:50.996966+0000","last_clean":"2024-04-22T08:34:50.996966+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:50.996966+0000","last_undegraded":"2024-04-22T08:34:50.996966+0000","last_fullsized":"2024-04-22T08:34:50.996966+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17145856,"num_objects":4186,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15777","reported_seq":15919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.579841+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:51.579841+0000","last_peered":"2024-04-22T08:34:51.579841+0000","last_clean":"2024-04-22T08:34:51.579841+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:51.579841+0000","last_undegraded":"2024-04-22T08:34:51.579841+0000","last_fullsized":"2024-04-22T08:34:51.579841+0000","mapping_epoch":27,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16965632,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12869,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'15951","reported_seq":23432,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.654245+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:51.654245+0000","last_peered":"2024-04-22T08:34:51.654245+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:51.654245+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4096,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12820,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3658,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15670","reported_seq":23410,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:50.699818+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:50.699818+0000","last_peered":"2024-04-22T08:34:50.699818+0000","last_clean":"2024-04-22T08:34:50.699818+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:50.699818+0000","last_undegraded":"2024-04-22T08:34:50.699818+0000","last_fullsized":"2024-04-22T08:34:50.699818+0000","mapping_epoch":36,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15670","reported_seq":15818,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.626940+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:51.626940+0000","last_peered":"2024-04-22T08:34:51.626940+0000","last_clean":"2024-04-22T08:34:51.626940+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:51.626940+0000","last_undegraded":"2024-04-22T08:34:51.626940+0000","last_fullsized":"2024-04-22T08:34:51.626940+0000","mapping_epoch":27,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16941056,"num_objects":4136,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16001","reported_seq":16458,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.095683+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:51.095683+0000","last_peered":"2024-04-22T08:34:51.095683+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:51.095683+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17286354,"num_objects":4222,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'15935","reported_seq":16816,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.465849+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:51.465849+0000","last_peered":"2024-04-22T08:34:51.465849+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:51.465849+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413330,"num_objects":4253,"num_object_clones":0,"num_object_copies":8506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15933,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":882}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15758","reported_seq":16290,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.053442+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:51.053442+0000","last_peered":"2024-04-22T08:34:51.053442+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:51.053442+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5700","ondisk_log_start":"35'5700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17084416,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12212,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15814","reported_seq":23492,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:51.076334+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:51.076334+0000","last_peered":"2024-04-22T08:34:51.076334+0000","last_clean":"2024-04-22T08:34:51.076334+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:51.076334+0000","last_undegraded":"2024-04-22T08:34:51.076334+0000","last_fullsized":"2024-04-22T08:34:51.076334+0000","mapping_epoch":38,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16876754,"num_objects":4122,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15812,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'15856","reported_seq":16608,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.601736+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:51.601736+0000","last_peered":"2024-04-22T08:34:51.601736+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:51.601736+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17556073,"num_objects":4287,"num_object_clones":0,"num_object_copies":8574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15855,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15677","reported_seq":16350,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.515481+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:51.515481+0000","last_peered":"2024-04-22T08:34:51.515481+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:51.515481+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5600","ondisk_log_start":"35'5600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16683008,"num_objects":4073,"num_object_clones":0,"num_object_copies":8146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15828","reported_seq":16430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.658118+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:51.658118+0000","last_peered":"2024-04-22T08:34:51.658118+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:51.658118+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17245394,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15826,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'15880","reported_seq":16592,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:51.617021+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:51.617021+0000","last_peered":"2024-04-22T08:34:51.617021+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:51.617021+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16886180,"num_objects":4126,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15877,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":273756013,"num_objects":66847,"num_object_clones":0,"num_object_copies":133694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226857,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735064064,"data_stored":734989819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117090,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584360,"kb_used_data":255768,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93787480,"statfs":{"total":96636764160,"available":96038379520,"internally_reserved":0,"allocated":261906432,"data_stored":261680505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336457423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.246},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.28000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621464,"kb_used_data":202640,"kb_used_omap":4,"kb_used_meta":418811,"kb_avail":93750376,"statfs":{"total":96636764160,"available":96000385024,"internally_reserved":0,"allocated":207503360,"data_stored":207221769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":428863180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":6.242}]}]},{"osd":0,"up_from":21,"seq":90194313318,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663000,"kb_used_data":262276,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93708840,"statfs":{"total":96636764160,"available":95957852160,"internally_reserved":0,"allocated":268570624,"data_stored":268355231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":410316499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":7.1790000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":6.7729999999999997},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267436032,"data_stored":267432553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206188544,"data_stored":206125329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261439488,"data_stored":261431937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:58.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:58.824 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-22T08:34:58.343990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1834032,"kb_used_data":702288,"kb_used_omap":12,"kb_used_meta":1131699,"kb_avail":281281488,"statfs":{"total":289910292480,"available":288032243712,"internally_reserved":0,"allocated":719142912,"data_stored":718420001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1158859886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,1,8,14,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-6070,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007570"},"pg_stats":[{"pgid":"2.f","version":"39'16143","reported_seq":16862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.692918+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:56.692918+0000","last_peered":"2024-04-22T08:34:56.692918+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:56.692918+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16486400,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16143,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":557}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'16091","reported_seq":16209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.069967+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:56.069967+0000","last_peered":"2024-04-22T08:34:56.069967+0000","last_clean":"2024-04-22T08:34:56.069967+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:56.069967+0000","last_undegraded":"2024-04-22T08:34:56.069967+0000","last_fullsized":"2024-04-22T08:34:56.069967+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16961536,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13147,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15831","reported_seq":16365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.114802+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:56.114802+0000","last_peered":"2024-04-22T08:34:56.114802+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:56.114802+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515096,"num_objects":4033,"num_object_clones":0,"num_object_copies":8066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12658,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'16078","reported_seq":16112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.070094+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:56.070094+0000","last_peered":"2024-04-22T08:34:56.070094+0000","last_clean":"2024-04-22T08:34:56.070094+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:56.070094+0000","last_undegraded":"2024-04-22T08:34:56.070094+0000","last_fullsized":"2024-04-22T08:34:56.070094+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16078,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15948","reported_seq":16090,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.686470+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:56.686470+0000","last_peered":"2024-04-22T08:34:56.686470+0000","last_clean":"2024-04-22T08:34:56.686470+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:56.686470+0000","last_undegraded":"2024-04-22T08:34:56.686470+0000","last_fullsized":"2024-04-22T08:34:56.686470+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16265216,"num_objects":3971,"num_object_clones":0,"num_object_copies":7942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13040,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'16104","reported_seq":23845,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.621573+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:56.621573+0000","last_peered":"2024-04-22T08:34:56.621573+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:56.621573+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16150528,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3788,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15820","reported_seq":23560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:55.701278+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:55.701278+0000","last_peered":"2024-04-22T08:34:55.701278+0000","last_clean":"2024-04-22T08:34:55.701278+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:55.701278+0000","last_undegraded":"2024-04-22T08:34:55.701278+0000","last_fullsized":"2024-04-22T08:34:55.701278+0000","mapping_epoch":36,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16564224,"num_objects":4044,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12188,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15807","reported_seq":15955,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.687709+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:56.687709+0000","last_peered":"2024-04-22T08:34:56.687709+0000","last_clean":"2024-04-22T08:34:56.687709+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:56.687709+0000","last_undegraded":"2024-04-22T08:34:56.687709+0000","last_fullsized":"2024-04-22T08:34:56.687709+0000","mapping_epoch":27,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16379904,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16151","reported_seq":16606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.132620+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:56.132620+0000","last_peered":"2024-04-22T08:34:56.132620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:56.132620+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16671954,"num_objects":4072,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'16073","reported_seq":16952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.705422+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:56.705422+0000","last_peered":"2024-04-22T08:34:56.705422+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:56.705422+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848082,"num_objects":4115,"num_object_clones":0,"num_object_copies":8230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":744}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15911","reported_seq":16443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.109234+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:56.109234+0000","last_peered":"2024-04-22T08:34:56.109234+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:56.109234+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16457728,"num_objects":4018,"num_object_clones":0,"num_object_copies":8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15945","reported_seq":23623,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.026853+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:56.026853+0000","last_peered":"2024-04-22T08:34:56.026853+0000","last_clean":"2024-04-22T08:34:56.026853+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:56.026853+0000","last_undegraded":"2024-04-22T08:34:56.026853+0000","last_fullsized":"2024-04-22T08:34:56.026853+0000","mapping_epoch":38,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15943,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'16007","reported_seq":16762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.646473+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:56.646473+0000","last_peered":"2024-04-22T08:34:56.646473+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:56.646473+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937577,"num_objects":4136,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16006,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15815","reported_seq":16488,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.647725+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:56.647725+0000","last_peered":"2024-04-22T08:34:56.647725+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:56.647725+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15985","reported_seq":16586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.597383+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:56.597383+0000","last_peered":"2024-04-22T08:34:56.597383+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:56.597383+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16602322,"num_objects":4055,"num_object_clones":0,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15983,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'16022","reported_seq":16736,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.670977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:56.670977+0000","last_peered":"2024-04-22T08:34:56.670977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:56.670977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16304548,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16019,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715829248,"data_stored":715755003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117091,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583844,"kb_used_data":251164,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93787996,"statfs":{"total":96636764160,"available":96038907904,"internally_reserved":0,"allocated":257191936,"data_stored":256966009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":6.2809999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.746},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":2.2730000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606380,"kb_used_data":195740,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93765460,"statfs":{"total":96636764160,"available":96015831040,"internally_reserved":0,"allocated":200437760,"data_stored":200156169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":420474572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.746},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.377},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313319,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643808,"kb_used_data":255384,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93728032,"statfs":{"total":96636764160,"available":95977504768,"internally_reserved":0,"allocated":261513216,"data_stored":261297823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":5.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260124672,"data_stored":260121193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199008256,"data_stored":198945041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256696320,"data_stored":256688769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:58.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:34:59.768 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:34:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:34:59.827 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-22T08:34:58.343990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1834032,"kb_used_data":702288,"kb_used_omap":12,"kb_used_meta":1131699,"kb_avail":281281488,"statfs":{"total":289910292480,"available":288032243712,"internally_reserved":0,"allocated":719142912,"data_stored":718420001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1158859886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,1,8,14,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-6070,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007570"},"pg_stats":[{"pgid":"2.f","version":"39'16143","reported_seq":16862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.692918+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:56.692918+0000","last_peered":"2024-04-22T08:34:56.692918+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:56.692918+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16486400,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16143,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":557}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'16091","reported_seq":16209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.069967+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:56.069967+0000","last_peered":"2024-04-22T08:34:56.069967+0000","last_clean":"2024-04-22T08:34:56.069967+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:56.069967+0000","last_undegraded":"2024-04-22T08:34:56.069967+0000","last_fullsized":"2024-04-22T08:34:56.069967+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16961536,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13147,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15831","reported_seq":16365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.114802+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:56.114802+0000","last_peered":"2024-04-22T08:34:56.114802+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:56.114802+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515096,"num_objects":4033,"num_object_clones":0,"num_object_copies":8066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12658,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'16078","reported_seq":16112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.070094+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:56.070094+0000","last_peered":"2024-04-22T08:34:56.070094+0000","last_clean":"2024-04-22T08:34:56.070094+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:56.070094+0000","last_undegraded":"2024-04-22T08:34:56.070094+0000","last_fullsized":"2024-04-22T08:34:56.070094+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16078,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15948","reported_seq":16090,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.686470+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:56.686470+0000","last_peered":"2024-04-22T08:34:56.686470+0000","last_clean":"2024-04-22T08:34:56.686470+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:56.686470+0000","last_undegraded":"2024-04-22T08:34:56.686470+0000","last_fullsized":"2024-04-22T08:34:56.686470+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16265216,"num_objects":3971,"num_object_clones":0,"num_object_copies":7942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13040,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'16104","reported_seq":23845,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.621573+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:56.621573+0000","last_peered":"2024-04-22T08:34:56.621573+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:56.621573+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16150528,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3788,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15820","reported_seq":23560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:55.701278+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:55.701278+0000","last_peered":"2024-04-22T08:34:55.701278+0000","last_clean":"2024-04-22T08:34:55.701278+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:55.701278+0000","last_undegraded":"2024-04-22T08:34:55.701278+0000","last_fullsized":"2024-04-22T08:34:55.701278+0000","mapping_epoch":36,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16564224,"num_objects":4044,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12188,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15807","reported_seq":15955,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.687709+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:56.687709+0000","last_peered":"2024-04-22T08:34:56.687709+0000","last_clean":"2024-04-22T08:34:56.687709+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:56.687709+0000","last_undegraded":"2024-04-22T08:34:56.687709+0000","last_fullsized":"2024-04-22T08:34:56.687709+0000","mapping_epoch":27,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16379904,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16151","reported_seq":16606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.132620+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:56.132620+0000","last_peered":"2024-04-22T08:34:56.132620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:56.132620+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16671954,"num_objects":4072,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'16073","reported_seq":16952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.705422+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:56.705422+0000","last_peered":"2024-04-22T08:34:56.705422+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:56.705422+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848082,"num_objects":4115,"num_object_clones":0,"num_object_copies":8230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":744}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15911","reported_seq":16443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.109234+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:56.109234+0000","last_peered":"2024-04-22T08:34:56.109234+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:56.109234+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16457728,"num_objects":4018,"num_object_clones":0,"num_object_copies":8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15945","reported_seq":23623,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.026853+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:56.026853+0000","last_peered":"2024-04-22T08:34:56.026853+0000","last_clean":"2024-04-22T08:34:56.026853+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:56.026853+0000","last_undegraded":"2024-04-22T08:34:56.026853+0000","last_fullsized":"2024-04-22T08:34:56.026853+0000","mapping_epoch":38,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15943,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'16007","reported_seq":16762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.646473+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:56.646473+0000","last_peered":"2024-04-22T08:34:56.646473+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:56.646473+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937577,"num_objects":4136,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16006,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15815","reported_seq":16488,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.647725+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:56.647725+0000","last_peered":"2024-04-22T08:34:56.647725+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:56.647725+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15985","reported_seq":16586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.597383+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:56.597383+0000","last_peered":"2024-04-22T08:34:56.597383+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:56.597383+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16602322,"num_objects":4055,"num_object_clones":0,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15983,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'16022","reported_seq":16736,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.670977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:56.670977+0000","last_peered":"2024-04-22T08:34:56.670977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:56.670977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16304548,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16019,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715829248,"data_stored":715755003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117091,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583844,"kb_used_data":251164,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93787996,"statfs":{"total":96636764160,"available":96038907904,"internally_reserved":0,"allocated":257191936,"data_stored":256966009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":6.2809999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.746},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":2.2730000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606380,"kb_used_data":195740,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93765460,"statfs":{"total":96636764160,"available":96015831040,"internally_reserved":0,"allocated":200437760,"data_stored":200156169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":420474572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.746},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.377},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313319,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643808,"kb_used_data":255384,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93728032,"statfs":{"total":96636764160,"available":95977504768,"internally_reserved":0,"allocated":261513216,"data_stored":261297823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":5.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260124672,"data_stored":260121193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199008256,"data_stored":198945041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256696320,"data_stored":256688769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:34:59.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:34:59.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'16143", 'reported_seq': 16862, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.692918+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:34:56.692918+0000', 'last_peered': '2024-04-22T08:34:56.692918+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:34:56.692918+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16486400, 'num_objects': 4025, 'num_object_clones': 0, 'num_object_copies': 8050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16143, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 557}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:34:59.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15831", 'reported_seq': 16365, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.114802+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:34:56.114802+0000', 'last_peered': '2024-04-22T08:34:56.114802+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:34:56.114802+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5800", 'ondisk_log_start': "35'5800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16515096, 'num_objects': 4033, 'num_object_clones': 0, 'num_object_copies': 8066, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4033, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12658, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'16104", 'reported_seq': 23845, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.621573+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:56.621573+0000', 'last_peered': '2024-04-22T08:34:56.621573+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:56.621573+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16150528, 'num_objects': 3943, 'num_object_clones': 0, 'num_object_copies': 7886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 94, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12973, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3788, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'16151", 'reported_seq': 16606, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.132620+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:34:56.132620+0000', 'last_peered': '2024-04-22T08:34:56.132620+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:34:56.132620+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16671954, 'num_objects': 4072, 'num_object_clones': 0, 'num_object_copies': 8144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16091, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'16073", 'reported_seq': 16952, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.705422+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:34:56.705422+0000', 'last_peered': '2024-04-22T08:34:56.705422+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:34:56.705422+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16848082, 'num_objects': 4115, 'num_object_clones': 0, 'num_object_copies': 8230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16071, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 744}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'15911", 'reported_seq': 16443, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.109234+0000', 'last_change': '2024-04-22T08:27:56.632887+0000', 'last_active': '2024-04-22T08:34:56.109234+0000', 'last_peered': '2024-04-22T08:34:56.109234+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:34:56.109234+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16457728, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 8036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12365, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 191, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 459}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:34:59.829 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'16007", 'reported_seq': 16762, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.646473+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:34:56.646473+0000', 'last_peered': '2024-04-22T08:34:56.646473+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:34:56.646473+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16937577, 'num_objects': 4136, 'num_object_clones': 0, 'num_object_copies': 8272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16006, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'15815", 'reported_seq': 16488, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.647725+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:34:56.647725+0000', 'last_peered': '2024-04-22T08:34:56.647725+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:34:56.647725+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'5800", 'ondisk_log_start': "35'5800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16117760, 'num_objects': 3935, 'num_object_clones': 0, 'num_object_copies': 7870, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12046, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'15985", 'reported_seq': 16586, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.597383+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:34:56.597383+0000', 'last_peered': '2024-04-22T08:34:56.597383+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:34:56.597383+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16602322, 'num_objects': 4055, 'num_object_clones': 0, 'num_object_copies': 8110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15983, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:34:59.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'16022", 'reported_seq': 16736, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.670977+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:34:56.670977+0000', 'last_peered': '2024-04-22T08:34:56.670977+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:34:56.670977+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16304548, 'num_objects': 3984, 'num_object_clones': 0, 'num_object_copies': 7968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16019, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:34:59.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:00.610 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31800/128996 objects degraded (24.652%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":535,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":64498,"data_bytes":264134509,"bytes_used":1878048768,"bytes_avail":288032243712,"bytes_total":289910292480,"degraded_objects":31800,"degraded_total":128996,"degraded_ratio":0.24651927191540823,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":265,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":394},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:00.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:00.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:01.509 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:01.559 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-04-22T08:35:00.346644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1834032,"kb_used_data":702288,"kb_used_omap":12,"kb_used_meta":1131699,"kb_avail":281281488,"statfs":{"total":289910292480,"available":288032243712,"internally_reserved":0,"allocated":719142912,"data_stored":718420001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1158859886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,1,8,14,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19419136,"num_objects":-4741,"num_object_clones":0,"num_object_copies":-9482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009750"},"pg_stats":[{"pgid":"2.f","version":"39'16143","reported_seq":16862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.692918+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:34:56.692918+0000","last_peered":"2024-04-22T08:34:56.692918+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:34:56.692918+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16486400,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16143,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":557}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'16091","reported_seq":16209,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.069967+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:34:56.069967+0000","last_peered":"2024-04-22T08:34:56.069967+0000","last_clean":"2024-04-22T08:34:56.069967+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:34:56.069967+0000","last_undegraded":"2024-04-22T08:34:56.069967+0000","last_fullsized":"2024-04-22T08:34:56.069967+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16961536,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13147,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15831","reported_seq":16365,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.114802+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:34:56.114802+0000","last_peered":"2024-04-22T08:34:56.114802+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:34:56.114802+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515096,"num_objects":4033,"num_object_clones":0,"num_object_copies":8066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12658,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'16078","reported_seq":16112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.070094+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:34:56.070094+0000","last_peered":"2024-04-22T08:34:56.070094+0000","last_clean":"2024-04-22T08:34:56.070094+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:34:56.070094+0000","last_undegraded":"2024-04-22T08:34:56.070094+0000","last_fullsized":"2024-04-22T08:34:56.070094+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16078,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'15948","reported_seq":16090,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.686470+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:34:56.686470+0000","last_peered":"2024-04-22T08:34:56.686470+0000","last_clean":"2024-04-22T08:34:56.686470+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:34:56.686470+0000","last_undegraded":"2024-04-22T08:34:56.686470+0000","last_fullsized":"2024-04-22T08:34:56.686470+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16265216,"num_objects":3971,"num_object_clones":0,"num_object_copies":7942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13040,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'16104","reported_seq":23845,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.621573+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:56.621573+0000","last_peered":"2024-04-22T08:34:56.621573+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:56.621573+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16150528,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3788,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15820","reported_seq":23560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:55.701278+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:34:55.701278+0000","last_peered":"2024-04-22T08:34:55.701278+0000","last_clean":"2024-04-22T08:34:55.701278+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:34:55.701278+0000","last_undegraded":"2024-04-22T08:34:55.701278+0000","last_fullsized":"2024-04-22T08:34:55.701278+0000","mapping_epoch":36,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16564224,"num_objects":4044,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12188,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"39'15807","reported_seq":15955,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.687709+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:34:56.687709+0000","last_peered":"2024-04-22T08:34:56.687709+0000","last_clean":"2024-04-22T08:34:56.687709+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:34:56.687709+0000","last_undegraded":"2024-04-22T08:34:56.687709+0000","last_fullsized":"2024-04-22T08:34:56.687709+0000","mapping_epoch":27,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16379904,"num_objects":3999,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16151","reported_seq":16606,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.132620+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:34:56.132620+0000","last_peered":"2024-04-22T08:34:56.132620+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:34:56.132620+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16671954,"num_objects":4072,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'16073","reported_seq":16952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.705422+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:34:56.705422+0000","last_peered":"2024-04-22T08:34:56.705422+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:34:56.705422+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848082,"num_objects":4115,"num_object_clones":0,"num_object_copies":8230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":744}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":71,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.138665+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:32:19.138665+0000","last_peered":"2024-04-22T08:32:19.138665+0000","last_clean":"2024-04-22T08:32:19.138665+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:32:19.138665+0000","last_undegraded":"2024-04-22T08:32:19.138665+0000","last_fullsized":"2024-04-22T08:32:19.138665+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'15911","reported_seq":16443,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.109234+0000","last_change":"2024-04-22T08:27:56.632887+0000","last_active":"2024-04-22T08:34:56.109234+0000","last_peered":"2024-04-22T08:34:56.109234+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:34:56.109234+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16457728,"num_objects":4018,"num_object_clones":0,"num_object_copies":8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'15945","reported_seq":23623,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:34:56.026853+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:34:56.026853+0000","last_peered":"2024-04-22T08:34:56.026853+0000","last_clean":"2024-04-22T08:34:56.026853+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:34:56.026853+0000","last_undegraded":"2024-04-22T08:34:56.026853+0000","last_fullsized":"2024-04-22T08:34:56.026853+0000","mapping_epoch":38,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15943,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'16007","reported_seq":16762,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.646473+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:34:56.646473+0000","last_peered":"2024-04-22T08:34:56.646473+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:34:56.646473+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937577,"num_objects":4136,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16006,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'15815","reported_seq":16488,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.647725+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:34:56.647725+0000","last_peered":"2024-04-22T08:34:56.647725+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:34:56.647725+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5800","ondisk_log_start":"35'5800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117760,"num_objects":3935,"num_object_clones":0,"num_object_copies":7870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'15985","reported_seq":16586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.597383+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:34:56.597383+0000","last_peered":"2024-04-22T08:34:56.597383+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:34:56.597383+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16602322,"num_objects":4055,"num_object_clones":0,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15983,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'16022","reported_seq":16736,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.670977+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:34:56.670977+0000","last_peered":"2024-04-22T08:34:56.670977+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:34:56.670977+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16304548,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16019,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":264134509,"num_objects":64498,"num_object_clones":0,"num_object_copies":128996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229206,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13807,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715829248,"data_stored":715755003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160631,"ondisk_log_size":160631,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117091,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583844,"kb_used_data":251164,"kb_used_omap":4,"kb_used_meta":332667,"kb_avail":93787996,"statfs":{"total":96636764160,"available":96038907904,"internally_reserved":0,"allocated":257191936,"data_stored":256966009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":340651727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":6.2809999999999997},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.746},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":2.2730000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606380,"kb_used_data":195740,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93765460,"statfs":{"total":96636764160,"available":96015831040,"internally_reserved":0,"allocated":200437760,"data_stored":200156169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":420474572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":1.746},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.377},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313319,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643808,"kb_used_data":255384,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93728032,"statfs":{"total":96636764160,"available":95977504768,"internally_reserved":0,"allocated":261513216,"data_stored":261297823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":397733587},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":5.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260124672,"data_stored":260121193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199008256,"data_stored":198945041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256696320,"data_stored":256688769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:04.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:05.320 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:05.322 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:05.375 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-04-22T08:35:04.348632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255139693,"num_objects":62302,"num_object_clones":0,"num_object_copies":124604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231402,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13811,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1833160,"kb_used_data":683372,"kb_used_omap":12,"kb_used_meta":1149747,"kb_avail":281282360,"statfs":{"total":289910292480,"available":288033136640,"internally_reserved":0,"allocated":699772928,"data_stored":699050017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1177341038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,10,11,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14471168,"num_objects":-3533,"num_object_clones":0,"num_object_copies":-7066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005150"},"pg_stats":[{"pgid":"2.f","version":"40'16309","reported_seq":17029,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.646123+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:01.646123+0000","last_peered":"2024-04-22T08:35:01.646123+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:01.646123+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16309,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'16248","reported_seq":16367,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.111974+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:01.111974+0000","last_peered":"2024-04-22T08:35:01.111974+0000","last_clean":"2024-04-22T08:35:01.111974+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:01.111974+0000","last_undegraded":"2024-04-22T08:35:01.111974+0000","last_fullsized":"2024-04-22T08:35:01.111974+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15983","reported_seq":16513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.156842+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:01.156842+0000","last_peered":"2024-04-22T08:35:01.156842+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:01.156842+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15892504,"num_objects":3881,"num_object_clones":0,"num_object_copies":7762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12810,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'16222","reported_seq":16256,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.005943+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:01.005943+0000","last_peered":"2024-04-22T08:35:01.005943+0000","last_clean":"2024-04-22T08:35:01.005943+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:01.005943+0000","last_undegraded":"2024-04-22T08:35:01.005943+0000","last_fullsized":"2024-04-22T08:35:01.005943+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15941632,"num_objects":3892,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16222,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"40'16091","reported_seq":16234,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.692406+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:01.692406+0000","last_peered":"2024-04-22T08:35:01.692406+0000","last_clean":"2024-04-22T08:35:01.692406+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:01.692406+0000","last_undegraded":"2024-04-22T08:35:01.692406+0000","last_fullsized":"2024-04-22T08:35:01.692406+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15679488,"num_objects":3828,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13183,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'16104","reported_seq":23845,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.621573+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:56.621573+0000","last_peered":"2024-04-22T08:34:56.621573+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:56.621573+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16150528,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3788,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15961","reported_seq":23701,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:00.784959+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:00.784959+0000","last_peered":"2024-04-22T08:35:00.784959+0000","last_clean":"2024-04-22T08:35:00.784959+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:00.784959+0000","last_undegraded":"2024-04-22T08:35:00.784959+0000","last_fullsized":"2024-04-22T08:35:00.784959+0000","mapping_epoch":36,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15986688,"num_objects":3903,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"40'15954","reported_seq":16103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.668903+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:01.668903+0000","last_peered":"2024-04-22T08:35:01.668903+0000","last_clean":"2024-04-22T08:35:01.668903+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:01.668903+0000","last_undegraded":"2024-04-22T08:35:01.668903+0000","last_fullsized":"2024-04-22T08:35:01.668903+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16292","reported_seq":16748,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.143248+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:01.143248+0000","last_peered":"2024-04-22T08:35:01.143248+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:01.143248+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094418,"num_objects":3931,"num_object_clones":0,"num_object_copies":7862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"40'16223","reported_seq":17104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.688843+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:01.688843+0000","last_peered":"2024-04-22T08:35:01.688843+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:01.688843+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233682,"num_objects":3965,"num_object_clones":0,"num_object_copies":7930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":72,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.861748+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:33:05.861748+0000","last_peered":"2024-04-22T08:33:05.861748+0000","last_clean":"2024-04-22T08:33:05.861748+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:33:05.861748+0000","last_undegraded":"2024-04-22T08:33:05.861748+0000","last_fullsized":"2024-04-22T08:33:05.861748+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'16055","reported_seq":16595,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.108733+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:01.108733+0000","last_peered":"2024-04-22T08:35:01.108733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:01.108733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15867904,"num_objects":3874,"num_object_clones":0,"num_object_copies":7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12509,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'16076","reported_seq":23754,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.095630+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:01.095630+0000","last_peered":"2024-04-22T08:35:01.095630+0000","last_clean":"2024-04-22T08:35:01.095630+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:01.095630+0000","last_undegraded":"2024-04-22T08:35:01.095630+0000","last_fullsized":"2024-04-22T08:35:01.095630+0000","mapping_epoch":38,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15803602,"num_objects":3860,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16074,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"40'16152","reported_seq":16908,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.645381+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:01.645381+0000","last_peered":"2024-04-22T08:35:01.645381+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:01.645381+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16343657,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16151,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'15937","reported_seq":16611,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.594795+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:01.594795+0000","last_peered":"2024-04-22T08:35:01.594795+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:01.594795+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"40'16143","reported_seq":16743,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.701693+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:01.701693+0000","last_peered":"2024-04-22T08:35:01.701693+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:01.701693+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955154,"num_objects":3897,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16141,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"40'16177","reported_seq":16892,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.658678+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:01.658678+0000","last_peered":"2024-04-22T08:35:01.658678+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:01.658678+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15669668,"num_objects":3829,"num_object_clones":0,"num_object_copies":7658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":255139693,"num_objects":62302,"num_object_clones":0,"num_object_copies":124604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231402,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13811,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696524800,"data_stored":696454034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117092,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566916,"kb_used_data":246524,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93804924,"statfs":{"total":96636764160,"available":96056242176,"internally_reserved":0,"allocated":252440576,"data_stored":252214649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":1.1359999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603388,"kb_used_data":188660,"kb_used_omap":4,"kb_used_meta":414715,"kb_avail":93768452,"statfs":{"total":96636764160,"available":96018894848,"internally_reserved":0,"allocated":193187840,"data_stored":192906249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":424668876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.27400000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313320,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662856,"kb_used_data":248188,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93708984,"statfs":{"total":96636764160,"available":95957999616,"internally_reserved":0,"allocated":254144512,"data_stored":253929119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.5540000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252895232,"data_stored":252891753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191717376,"data_stored":191657640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251912192,"data_stored":251904641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:05.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:06.113 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:06.114 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:06.185 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-04-22T08:35:04.348632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255139693,"num_objects":62302,"num_object_clones":0,"num_object_copies":124604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231402,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13811,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1833160,"kb_used_data":683372,"kb_used_omap":12,"kb_used_meta":1149747,"kb_avail":281282360,"statfs":{"total":289910292480,"available":288033136640,"internally_reserved":0,"allocated":699772928,"data_stored":699050017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1177341038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,10,11,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14471168,"num_objects":-3533,"num_object_clones":0,"num_object_copies":-7066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005150"},"pg_stats":[{"pgid":"2.f","version":"40'16309","reported_seq":17029,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.646123+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:01.646123+0000","last_peered":"2024-04-22T08:35:01.646123+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:01.646123+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16309,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'16248","reported_seq":16367,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.111974+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:01.111974+0000","last_peered":"2024-04-22T08:35:01.111974+0000","last_clean":"2024-04-22T08:35:01.111974+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:01.111974+0000","last_undegraded":"2024-04-22T08:35:01.111974+0000","last_fullsized":"2024-04-22T08:35:01.111974+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'15983","reported_seq":16513,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.156842+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:01.156842+0000","last_peered":"2024-04-22T08:35:01.156842+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:01.156842+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15892504,"num_objects":3881,"num_object_clones":0,"num_object_copies":7762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12810,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'16222","reported_seq":16256,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.005943+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:01.005943+0000","last_peered":"2024-04-22T08:35:01.005943+0000","last_clean":"2024-04-22T08:35:01.005943+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:01.005943+0000","last_undegraded":"2024-04-22T08:35:01.005943+0000","last_fullsized":"2024-04-22T08:35:01.005943+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15941632,"num_objects":3892,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16222,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"40'16091","reported_seq":16234,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.692406+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:01.692406+0000","last_peered":"2024-04-22T08:35:01.692406+0000","last_clean":"2024-04-22T08:35:01.692406+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:01.692406+0000","last_undegraded":"2024-04-22T08:35:01.692406+0000","last_fullsized":"2024-04-22T08:35:01.692406+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15679488,"num_objects":3828,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13183,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'16104","reported_seq":23845,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:34:56.621573+0000","last_change":"2024-04-22T08:32:17.676737+0000","last_active":"2024-04-22T08:34:56.621573+0000","last_peered":"2024-04-22T08:34:56.621573+0000","last_clean":"2024-04-22T08:27:54.052724+0000","last_became_active":"2024-04-22T08:27:56.582379+0000","last_became_peered":"2024-04-22T08:27:56.582379+0000","last_unstale":"2024-04-22T08:34:56.621573+0000","last_undegraded":"2024-04-22T08:27:56.212564+0000","last_fullsized":"2024-04-22T08:27:56.084951+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16150528,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3788,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'15961","reported_seq":23701,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:00.784959+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:00.784959+0000","last_peered":"2024-04-22T08:35:00.784959+0000","last_clean":"2024-04-22T08:35:00.784959+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:00.784959+0000","last_undegraded":"2024-04-22T08:35:00.784959+0000","last_fullsized":"2024-04-22T08:35:00.784959+0000","mapping_epoch":36,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15986688,"num_objects":3903,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"40'15954","reported_seq":16103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.668903+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:01.668903+0000","last_peered":"2024-04-22T08:35:01.668903+0000","last_clean":"2024-04-22T08:35:01.668903+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:01.668903+0000","last_undegraded":"2024-04-22T08:35:01.668903+0000","last_fullsized":"2024-04-22T08:35:01.668903+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.140454+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:32:19.140454+0000","last_peered":"2024-04-22T08:32:19.140454+0000","last_clean":"2024-04-22T08:32:19.140454+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:32:19.140454+0000","last_undegraded":"2024-04-22T08:32:19.140454+0000","last_fullsized":"2024-04-22T08:32:19.140454+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'16292","reported_seq":16748,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.143248+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:01.143248+0000","last_peered":"2024-04-22T08:35:01.143248+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:01.143248+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094418,"num_objects":3931,"num_object_clones":0,"num_object_copies":7862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143592+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:32:19.143592+0000","last_peered":"2024-04-22T08:32:19.143592+0000","last_clean":"2024-04-22T08:32:19.143592+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:32:19.143592+0000","last_undegraded":"2024-04-22T08:32:19.143592+0000","last_fullsized":"2024-04-22T08:32:19.143592+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"40'16223","reported_seq":17104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.688843+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:01.688843+0000","last_peered":"2024-04-22T08:35:01.688843+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:01.688843+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233682,"num_objects":3965,"num_object_clones":0,"num_object_copies":7930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":72,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.861748+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:33:05.861748+0000","last_peered":"2024-04-22T08:33:05.861748+0000","last_clean":"2024-04-22T08:33:05.861748+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:33:05.861748+0000","last_undegraded":"2024-04-22T08:33:05.861748+0000","last_fullsized":"2024-04-22T08:33:05.861748+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'16055","reported_seq":16595,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.108733+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:01.108733+0000","last_peered":"2024-04-22T08:35:01.108733+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:01.108733+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15867904,"num_objects":3874,"num_object_clones":0,"num_object_copies":7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12509,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142870+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:32:19.142870+0000","last_peered":"2024-04-22T08:32:19.142870+0000","last_clean":"2024-04-22T08:32:19.142870+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:32:19.142870+0000","last_undegraded":"2024-04-22T08:32:19.142870+0000","last_fullsized":"2024-04-22T08:32:19.142870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'16076","reported_seq":23754,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.095630+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:01.095630+0000","last_peered":"2024-04-22T08:35:01.095630+0000","last_clean":"2024-04-22T08:35:01.095630+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:01.095630+0000","last_undegraded":"2024-04-22T08:35:01.095630+0000","last_fullsized":"2024-04-22T08:35:01.095630+0000","mapping_epoch":38,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15803602,"num_objects":3860,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16074,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"40'16152","reported_seq":16908,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.645381+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:01.645381+0000","last_peered":"2024-04-22T08:35:01.645381+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:01.645381+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16343657,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16151,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.142956+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:32:19.142956+0000","last_peered":"2024-04-22T08:32:19.142956+0000","last_clean":"2024-04-22T08:32:19.142956+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:32:19.142956+0000","last_undegraded":"2024-04-22T08:32:19.142956+0000","last_fullsized":"2024-04-22T08:32:19.142956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'15937","reported_seq":16611,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.594795+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:01.594795+0000","last_peered":"2024-04-22T08:35:01.594795+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:01.594795+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.145830+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:32:19.145830+0000","last_peered":"2024-04-22T08:32:19.145830+0000","last_clean":"2024-04-22T08:32:19.145830+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:32:19.145830+0000","last_undegraded":"2024-04-22T08:32:19.145830+0000","last_fullsized":"2024-04-22T08:32:19.145830+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"40'16143","reported_seq":16743,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.701693+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:01.701693+0000","last_peered":"2024-04-22T08:35:01.701693+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:01.701693+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955154,"num_objects":3897,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16141,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.144937+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:32:19.144937+0000","last_peered":"2024-04-22T08:32:19.144937+0000","last_clean":"2024-04-22T08:32:19.144937+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:32:19.144937+0000","last_undegraded":"2024-04-22T08:32:19.144937+0000","last_fullsized":"2024-04-22T08:32:19.144937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:32:19.143567+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:32:19.143567+0000","last_peered":"2024-04-22T08:32:19.143567+0000","last_clean":"2024-04-22T08:32:19.143567+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:32:19.143567+0000","last_undegraded":"2024-04-22T08:32:19.143567+0000","last_fullsized":"2024-04-22T08:32:19.143567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"40'16177","reported_seq":16892,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.658678+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:01.658678+0000","last_peered":"2024-04-22T08:35:01.658678+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:01.658678+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15669668,"num_objects":3829,"num_object_clones":0,"num_object_copies":7658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":255139693,"num_objects":62302,"num_object_clones":0,"num_object_copies":124604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231402,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13811,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696524800,"data_stored":696454034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117092,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566916,"kb_used_data":246524,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93804924,"statfs":{"total":96636764160,"available":96056242176,"internally_reserved":0,"allocated":252440576,"data_stored":252214649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":1.1359999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603388,"kb_used_data":188660,"kb_used_omap":4,"kb_used_meta":414715,"kb_avail":93768452,"statfs":{"total":96636764160,"available":96018894848,"internally_reserved":0,"allocated":193187840,"data_stored":192906249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":424668876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.27400000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313320,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662856,"kb_used_data":248188,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93708984,"statfs":{"total":96636764160,"available":95957999616,"internally_reserved":0,"allocated":254144512,"data_stored":253929119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.5540000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252895232,"data_stored":252891753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191717376,"data_stored":191657640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":251912192,"data_stored":251904641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:06.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:06.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "40'16309", 'reported_seq': 17029, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.646123+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:01.646123+0000', 'last_peered': '2024-04-22T08:35:01.646123+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:01.646123+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6300", 'ondisk_log_start': "35'6300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15806464, 'num_objects': 3859, 'num_object_clones': 0, 'num_object_copies': 7718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16309, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 391}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:06.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'15983", 'reported_seq': 16513, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.156842+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:01.156842+0000', 'last_peered': '2024-04-22T08:35:01.156842+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:01.156842+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15892504, 'num_objects': 3881, 'num_object_clones': 0, 'num_object_copies': 7762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3881, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12810, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:06.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'16104", 'reported_seq': 23845, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:34:56.621573+0000', 'last_change': '2024-04-22T08:32:17.676737+0000', 'last_active': '2024-04-22T08:34:56.621573+0000', 'last_peered': '2024-04-22T08:34:56.621573+0000', 'last_clean': '2024-04-22T08:27:54.052724+0000', 'last_became_active': '2024-04-22T08:27:56.582379+0000', 'last_became_peered': '2024-04-22T08:27:56.582379+0000', 'last_unstale': '2024-04-22T08:34:56.621573+0000', 'last_undegraded': '2024-04-22T08:27:56.212564+0000', 'last_fullsized': '2024-04-22T08:27:56.084951+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3186", 'last_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:42.225197+0000', 'objects_scrubbed': 3126, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:09:02.530751+0000', 'scrub_duration': 1.6537587440000001, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16150528, 'num_objects': 3943, 'num_object_clones': 0, 'num_object_copies': 7886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 94, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12973, 'num_write_kb': 27580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3788, 'num_bytes_recovered': 7528448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'16292", 'reported_seq': 16748, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.143248+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:01.143248+0000', 'last_peered': '2024-04-22T08:35:01.143248+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:01.143248+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6200", 'ondisk_log_start': "35'6200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16094418, 'num_objects': 3931, 'num_object_clones': 0, 'num_object_copies': 7862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16232, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "40'16223", 'reported_seq': 17104, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.688843+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:01.688843+0000', 'last_peered': '2024-04-22T08:35:01.688843+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:01.688843+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6200", 'ondisk_log_start': "35'6200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16233682, 'num_objects': 3965, 'num_object_clones': 0, 'num_object_copies': 7930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16221, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 594}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'16055", 'reported_seq': 16595, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.108733+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:01.108733+0000', 'last_peered': '2024-04-22T08:35:01.108733+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:01.108733+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15867904, 'num_objects': 3874, 'num_object_clones': 0, 'num_object_copies': 7748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3557, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12509, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 195, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 317}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "40'16152", 'reported_seq': 16908, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.645381+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:01.645381+0000', 'last_peered': '2024-04-22T08:35:01.645381+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:01.645381+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16343657, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 7982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16151, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:06.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "40'15937", 'reported_seq': 16611, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.594795+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:01.594795+0000', 'last_peered': '2024-04-22T08:35:01.594795+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:01.594795+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'5900", 'ondisk_log_start': "35'5900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15618048, 'num_objects': 3813, 'num_object_clones': 0, 'num_object_copies': 7626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12168, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:06.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "40'16143", 'reported_seq': 16743, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.701693+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:01.701693+0000', 'last_peered': '2024-04-22T08:35:01.701693+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:01.701693+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15955154, 'num_objects': 3897, 'num_object_clones': 0, 'num_object_copies': 7794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16141, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:06.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "40'16177", 'reported_seq': 16892, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:01.658678+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:01.658678+0000', 'last_peered': '2024-04-22T08:35:01.658678+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:01.658678+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15669668, 'num_objects': 3829, 'num_object_clones': 0, 'num_object_copies': 7658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16174, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:06.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31555/122202 objects degraded (25.822%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":542,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":61101,"data_bytes":250220397,"bytes_used":1880674304,"bytes_avail":288029618176,"bytes_total":289910292480,"degraded_objects":31555,"degraded_total":122202,"degraded_ratio":0.25821999639940424,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":424},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:07.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:07.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:07.712 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:07.713 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:07.752 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-22T08:35:06.349071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250220397,"num_objects":61101,"num_object_clones":0,"num_object_copies":122202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232603,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1836596,"kb_used_data":672824,"kb_used_omap":12,"kb_used_meta":1163699,"kb_avail":281278924,"statfs":{"total":289910292480,"available":288029618176,"internally_reserved":0,"allocated":688971776,"data_stored":688234833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1191627886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,9,14,21,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13914112,"num_objects":-3397,"num_object_clones":0,"num_object_copies":-6794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005081"},"pg_stats":[{"pgid":"2.f","version":"40'16309","reported_seq":17029,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.646123+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:01.646123+0000","last_peered":"2024-04-22T08:35:01.646123+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:01.646123+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16309,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":391}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16397","reported_seq":16518,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.155093+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:06.155093+0000","last_peered":"2024-04-22T08:35:06.155093+0000","last_clean":"2024-04-22T08:35:06.155093+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:06.155093+0000","last_undegraded":"2024-04-22T08:35:06.155093+0000","last_fullsized":"2024-04-22T08:35:06.155093+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15708160,"num_objects":3835,"num_object_clones":0,"num_object_copies":7670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16118","reported_seq":16654,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.084817+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:06.084817+0000","last_peered":"2024-04-22T08:35:06.084817+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:06.084817+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15339544,"num_objects":3746,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12945,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16355","reported_seq":16391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.190590+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:06.190590+0000","last_peered":"2024-04-22T08:35:06.190590+0000","last_clean":"2024-04-22T08:35:06.190590+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:06.190590+0000","last_undegraded":"2024-04-22T08:35:06.190590+0000","last_fullsized":"2024-04-22T08:35:06.190590+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15396864,"num_objects":3759,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16355,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"40'16091","reported_seq":16234,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.692406+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:01.692406+0000","last_peered":"2024-04-22T08:35:01.692406+0000","last_clean":"2024-04-22T08:35:01.692406+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:01.692406+0000","last_undegraded":"2024-04-22T08:35:01.692406+0000","last_fullsized":"2024-04-22T08:35:01.692406+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15679488,"num_objects":3828,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13183,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16356","reported_seq":24298,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.896201+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:05.896201+0000","last_peered":"2024-04-22T08:35:05.896201+0000","last_clean":"2024-04-22T08:35:05.896201+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:05.896201+0000","last_undegraded":"2024-04-22T08:35:05.896201+0000","last_fullsized":"2024-04-22T08:35:05.896201+0000","mapping_epoch":40,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15118336,"num_objects":3691,"num_object_clones":0,"num_object_copies":7382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13225,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16088","reported_seq":23830,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.860698+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:05.860698+0000","last_peered":"2024-04-22T08:35:05.860698+0000","last_clean":"2024-04-22T08:35:05.860698+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:05.860698+0000","last_undegraded":"2024-04-22T08:35:05.860698+0000","last_fullsized":"2024-04-22T08:35:05.860698+0000","mapping_epoch":36,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"40'15954","reported_seq":16103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.668903+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:01.668903+0000","last_peered":"2024-04-22T08:35:01.668903+0000","last_clean":"2024-04-22T08:35:01.668903+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:01.668903+0000","last_undegraded":"2024-04-22T08:35:01.668903+0000","last_fullsized":"2024-04-22T08:35:01.668903+0000","mapping_epoch":27,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16432","reported_seq":16891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.190582+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:06.190582+0000","last_peered":"2024-04-22T08:35:06.190582+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:06.190582+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15520978,"num_objects":3791,"num_object_clones":0,"num_object_copies":7582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16372,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":68}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"40'16223","reported_seq":17104,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.688843+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:01.688843+0000","last_peered":"2024-04-22T08:35:01.688843+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:01.688843+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16233682,"num_objects":3965,"num_object_clones":0,"num_object_copies":7930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":72,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T08:33:05.861748+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:33:05.861748+0000","last_peered":"2024-04-22T08:33:05.861748+0000","last_clean":"2024-04-22T08:33:05.861748+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:33:05.861748+0000","last_undegraded":"2024-04-22T08:33:05.861748+0000","last_fullsized":"2024-04-22T08:33:05.861748+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16187","reported_seq":17024,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.192058+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:06.192058+0000","last_peered":"2024-04-22T08:35:06.192058+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:06.192058+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15327232,"num_objects":3742,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16209","reported_seq":23890,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.183823+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:06.183823+0000","last_peered":"2024-04-22T08:35:06.183823+0000","last_clean":"2024-04-22T08:35:06.183823+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:06.183823+0000","last_undegraded":"2024-04-22T08:35:06.183823+0000","last_fullsized":"2024-04-22T08:35:06.183823+0000","mapping_epoch":38,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15258834,"num_objects":3727,"num_object_clones":0,"num_object_copies":7454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16207,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"40'16152","reported_seq":16908,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.645381+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:01.645381+0000","last_peered":"2024-04-22T08:35:01.645381+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:01.645381+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16343657,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16151,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'15937","reported_seq":16611,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.594795+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:01.594795+0000","last_peered":"2024-04-22T08:35:01.594795+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:01.594795+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'5900","ondisk_log_start":"35'5900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618048,"num_objects":3813,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"40'16143","reported_seq":16743,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.701693+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:01.701693+0000","last_peered":"2024-04-22T08:35:01.701693+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:01.701693+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955154,"num_objects":3897,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16141,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"40'16177","reported_seq":16892,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:01.658678+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:01.658678+0000","last_peered":"2024-04-22T08:35:01.658678+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:01.658678+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15669668,"num_objects":3829,"num_object_clones":0,"num_object_copies":7658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":250220397,"num_objects":61101,"num_object_clones":0,"num_object_copies":122202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232603,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":685432832,"data_stored":685365545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117093,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584644,"kb_used_data":242104,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93787196,"statfs":{"total":96636764160,"available":96038088704,"internally_reserved":0,"allocated":247914496,"data_stored":247678074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.095},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.59199999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589096,"kb_used_data":182532,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93782744,"statfs":{"total":96636764160,"available":96033529856,"internally_reserved":0,"allocated":186912768,"data_stored":186627640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":416280268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.81000000000000005}]}]},{"osd":0,"up_from":21,"seq":90194313320,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662856,"kb_used_data":248188,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93708984,"statfs":{"total":96636764160,"available":95957999616,"internally_reserved":0,"allocated":254144512,"data_stored":253929119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.4390000000000001,"15min":0.69099999999999995},"min":{"1min":0.19800000000000001,"5min":0.16,"15min":0.16},"max":{"1min":4.5510000000000002,"5min":9.3770000000000007,"15min":9.3770000000000007},"last":4.5540000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":1.087,"15min":0.94699999999999995},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.367,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252895232,"data_stored":252891753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":185176064,"data_stored":185119807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247361536,"data_stored":247353985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:10.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:11.402 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:11.403 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:11.470 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-22T08:35:10.350055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245837677,"num_objects":60031,"num_object_clones":0,"num_object_copies":120062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233673,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160898,"ondisk_log_size":160898,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1834260,"kb_used_data":666396,"kb_used_omap":12,"kb_used_meta":1167795,"kb_avail":281281260,"statfs":{"total":289910292480,"available":288032010240,"internally_reserved":0,"allocated":682389504,"data_stored":681642066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1195822190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,8,6,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14159872,"num_objects":-3457,"num_object_clones":0,"num_object_copies":-6914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.775087"},"pg_stats":[{"pgid":"2.f","version":"41'16433","reported_seq":17154,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.692717+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:06.692717+0000","last_peered":"2024-04-22T08:35:06.692717+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:06.692717+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15298560,"num_objects":3735,"num_object_clones":0,"num_object_copies":7470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16433,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16397","reported_seq":16518,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.155093+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:06.155093+0000","last_peered":"2024-04-22T08:35:06.155093+0000","last_clean":"2024-04-22T08:35:06.155093+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:06.155093+0000","last_undegraded":"2024-04-22T08:35:06.155093+0000","last_fullsized":"2024-04-22T08:35:06.155093+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15708160,"num_objects":3835,"num_object_clones":0,"num_object_copies":7670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16118","reported_seq":16654,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.084817+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:06.084817+0000","last_peered":"2024-04-22T08:35:06.084817+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:06.084817+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15339544,"num_objects":3746,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12945,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16355","reported_seq":16391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.190590+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:06.190590+0000","last_peered":"2024-04-22T08:35:06.190590+0000","last_clean":"2024-04-22T08:35:06.190590+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:06.190590+0000","last_undegraded":"2024-04-22T08:35:06.190590+0000","last_fullsized":"2024-04-22T08:35:06.190590+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15396864,"num_objects":3759,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16355,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16232","reported_seq":16376,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.631276+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:06.631276+0000","last_peered":"2024-04-22T08:35:06.631276+0000","last_clean":"2024-04-22T08:35:06.631276+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:06.631276+0000","last_undegraded":"2024-04-22T08:35:06.631276+0000","last_fullsized":"2024-04-22T08:35:06.631276+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15101952,"num_objects":3687,"num_object_clones":0,"num_object_copies":7374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16356","reported_seq":24298,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.896201+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:05.896201+0000","last_peered":"2024-04-22T08:35:05.896201+0000","last_clean":"2024-04-22T08:35:05.896201+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:05.896201+0000","last_undegraded":"2024-04-22T08:35:05.896201+0000","last_fullsized":"2024-04-22T08:35:05.896201+0000","mapping_epoch":40,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15118336,"num_objects":3691,"num_object_clones":0,"num_object_copies":7382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13225,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16088","reported_seq":23830,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.860698+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:05.860698+0000","last_peered":"2024-04-22T08:35:05.860698+0000","last_clean":"2024-04-22T08:35:05.860698+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:05.860698+0000","last_undegraded":"2024-04-22T08:35:05.860698+0000","last_fullsized":"2024-04-22T08:35:05.860698+0000","mapping_epoch":36,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16085","reported_seq":16235,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.708842+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:06.708842+0000","last_peered":"2024-04-22T08:35:06.708842+0000","last_clean":"2024-04-22T08:35:06.708842+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:06.708842+0000","last_undegraded":"2024-04-22T08:35:06.708842+0000","last_fullsized":"2024-04-22T08:35:06.708842+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15241216,"num_objects":3721,"num_object_clones":0,"num_object_copies":7442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16432","reported_seq":16891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.190582+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:06.190582+0000","last_peered":"2024-04-22T08:35:06.190582+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:06.190582+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15520978,"num_objects":3791,"num_object_clones":0,"num_object_copies":7582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16372,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":68}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16363","reported_seq":17246,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.715682+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:06.715682+0000","last_peered":"2024-04-22T08:35:06.715682+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:06.715682+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15660242,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16361,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":454}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.594489+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:01.594489+0000","last_peered":"2024-04-22T08:35:01.594489+0000","last_clean":"2024-04-22T08:35:01.594489+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:01.594489+0000","last_undegraded":"2024-04-22T08:35:01.594489+0000","last_fullsized":"2024-04-22T08:35:01.594489+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16187","reported_seq":17024,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.192058+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:06.192058+0000","last_peered":"2024-04-22T08:35:06.192058+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:06.192058+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15327232,"num_objects":3742,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16209","reported_seq":23890,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.183823+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:06.183823+0000","last_peered":"2024-04-22T08:35:06.183823+0000","last_clean":"2024-04-22T08:35:06.183823+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:06.183823+0000","last_undegraded":"2024-04-22T08:35:06.183823+0000","last_fullsized":"2024-04-22T08:35:06.183823+0000","mapping_epoch":38,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15258834,"num_objects":3727,"num_object_clones":0,"num_object_copies":7454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16207,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16287","reported_seq":17039,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.661647+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:06.661647+0000","last_peered":"2024-04-22T08:35:06.661647+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:06.661647+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15790697,"num_objects":3856,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16286,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16062","reported_seq":16733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.756646+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:06.756646+0000","last_peered":"2024-04-22T08:35:06.756646+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:06.756646+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15106048,"num_objects":3688,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":98}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16286","reported_seq":16890,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.671394+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:06.671394+0000","last_peered":"2024-04-22T08:35:06.671394+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:06.671394+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15369426,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16284,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16308","reported_seq":17024,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.709011+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:06.709011+0000","last_peered":"2024-04-22T08:35:06.709011+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:06.709011+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15133092,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16305,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":245837677,"num_objects":60031,"num_object_clones":0,"num_object_copies":120062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233673,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678830080,"data_stored":678762793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160898,"ondisk_log_size":160898,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117093,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584644,"kb_used_data":242104,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93787196,"statfs":{"total":96636764160,"available":96038088704,"internally_reserved":0,"allocated":247914496,"data_stored":247678074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.095},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.59199999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589096,"kb_used_data":182532,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93782744,"statfs":{"total":96636764160,"available":96033529856,"internally_reserved":0,"allocated":186912768,"data_stored":186627640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":416280268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.81000000000000005}]}]},{"osd":0,"up_from":21,"seq":90194313321,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660520,"kb_used_data":241760,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93711320,"statfs":{"total":96636764160,"available":95960391680,"internally_reserved":0,"allocated":247562240,"data_stored":247336352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":428797651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.317},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":7.4400000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246292480,"data_stored":246289001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":185176064,"data_stored":185119807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247361536,"data_stored":247353985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:11.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:12.247 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-22T08:35:10.350055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245837677,"num_objects":60031,"num_object_clones":0,"num_object_copies":120062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233673,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160898,"ondisk_log_size":160898,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1834260,"kb_used_data":666396,"kb_used_omap":12,"kb_used_meta":1167795,"kb_avail":281281260,"statfs":{"total":289910292480,"available":288032010240,"internally_reserved":0,"allocated":682389504,"data_stored":681642066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1195822190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,8,6,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14159872,"num_objects":-3457,"num_object_clones":0,"num_object_copies":-6914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.775087"},"pg_stats":[{"pgid":"2.f","version":"41'16433","reported_seq":17154,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.692717+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:06.692717+0000","last_peered":"2024-04-22T08:35:06.692717+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:06.692717+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15298560,"num_objects":3735,"num_object_clones":0,"num_object_copies":7470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16433,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16397","reported_seq":16518,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.155093+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:06.155093+0000","last_peered":"2024-04-22T08:35:06.155093+0000","last_clean":"2024-04-22T08:35:06.155093+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:06.155093+0000","last_undegraded":"2024-04-22T08:35:06.155093+0000","last_fullsized":"2024-04-22T08:35:06.155093+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15708160,"num_objects":3835,"num_object_clones":0,"num_object_copies":7670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16118","reported_seq":16654,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.084817+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:06.084817+0000","last_peered":"2024-04-22T08:35:06.084817+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:06.084817+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15339544,"num_objects":3746,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12945,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16355","reported_seq":16391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.190590+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:06.190590+0000","last_peered":"2024-04-22T08:35:06.190590+0000","last_clean":"2024-04-22T08:35:06.190590+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:06.190590+0000","last_undegraded":"2024-04-22T08:35:06.190590+0000","last_fullsized":"2024-04-22T08:35:06.190590+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15396864,"num_objects":3759,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16355,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16232","reported_seq":16376,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.631276+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:06.631276+0000","last_peered":"2024-04-22T08:35:06.631276+0000","last_clean":"2024-04-22T08:35:06.631276+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:06.631276+0000","last_undegraded":"2024-04-22T08:35:06.631276+0000","last_fullsized":"2024-04-22T08:35:06.631276+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15101952,"num_objects":3687,"num_object_clones":0,"num_object_copies":7374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16356","reported_seq":24298,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.896201+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:05.896201+0000","last_peered":"2024-04-22T08:35:05.896201+0000","last_clean":"2024-04-22T08:35:05.896201+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:05.896201+0000","last_undegraded":"2024-04-22T08:35:05.896201+0000","last_fullsized":"2024-04-22T08:35:05.896201+0000","mapping_epoch":40,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15118336,"num_objects":3691,"num_object_clones":0,"num_object_copies":7382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13225,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16088","reported_seq":23830,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:05.860698+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:05.860698+0000","last_peered":"2024-04-22T08:35:05.860698+0000","last_clean":"2024-04-22T08:35:05.860698+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:05.860698+0000","last_undegraded":"2024-04-22T08:35:05.860698+0000","last_fullsized":"2024-04-22T08:35:05.860698+0000","mapping_epoch":36,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16085","reported_seq":16235,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.708842+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:06.708842+0000","last_peered":"2024-04-22T08:35:06.708842+0000","last_clean":"2024-04-22T08:35:06.708842+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:06.708842+0000","last_undegraded":"2024-04-22T08:35:06.708842+0000","last_fullsized":"2024-04-22T08:35:06.708842+0000","mapping_epoch":27,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15241216,"num_objects":3721,"num_object_clones":0,"num_object_copies":7442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16432","reported_seq":16891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.190582+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:06.190582+0000","last_peered":"2024-04-22T08:35:06.190582+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:06.190582+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15520978,"num_objects":3791,"num_object_clones":0,"num_object_copies":7582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16372,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":68}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16363","reported_seq":17246,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.715682+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:06.715682+0000","last_peered":"2024-04-22T08:35:06.715682+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:06.715682+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15660242,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16361,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":454}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.594489+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:01.594489+0000","last_peered":"2024-04-22T08:35:01.594489+0000","last_clean":"2024-04-22T08:35:01.594489+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:01.594489+0000","last_undegraded":"2024-04-22T08:35:01.594489+0000","last_fullsized":"2024-04-22T08:35:01.594489+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16187","reported_seq":17024,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.192058+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:06.192058+0000","last_peered":"2024-04-22T08:35:06.192058+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:06.192058+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15327232,"num_objects":3742,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16209","reported_seq":23890,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:06.183823+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:06.183823+0000","last_peered":"2024-04-22T08:35:06.183823+0000","last_clean":"2024-04-22T08:35:06.183823+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:06.183823+0000","last_undegraded":"2024-04-22T08:35:06.183823+0000","last_fullsized":"2024-04-22T08:35:06.183823+0000","mapping_epoch":38,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15258834,"num_objects":3727,"num_object_clones":0,"num_object_copies":7454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16207,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16287","reported_seq":17039,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.661647+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:06.661647+0000","last_peered":"2024-04-22T08:35:06.661647+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:06.661647+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15790697,"num_objects":3856,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16286,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16062","reported_seq":16733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.756646+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:06.756646+0000","last_peered":"2024-04-22T08:35:06.756646+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:06.756646+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6000","ondisk_log_start":"35'6000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15106048,"num_objects":3688,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":98}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16286","reported_seq":16890,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.671394+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:06.671394+0000","last_peered":"2024-04-22T08:35:06.671394+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:06.671394+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15369426,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16284,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16308","reported_seq":17024,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:06.709011+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:06.709011+0000","last_peered":"2024-04-22T08:35:06.709011+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:06.709011+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15133092,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16305,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":245837677,"num_objects":60031,"num_object_clones":0,"num_object_copies":120062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233673,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14052,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678830080,"data_stored":678762793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160898,"ondisk_log_size":160898,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117093,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584644,"kb_used_data":242104,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93787196,"statfs":{"total":96636764160,"available":96038088704,"internally_reserved":0,"allocated":247914496,"data_stored":247678074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.4319999999999999,"15min":1.101},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.699,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.095},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.4790000000000001,"15min":0.876},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":7.556,"5min":18.32,"15min":18.32},"last":0.59199999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589096,"kb_used_data":182532,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93782744,"statfs":{"total":96636764160,"available":96033529856,"internally_reserved":0,"allocated":186912768,"data_stored":186627640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":416280268},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.84999999999999998,"15min":0.61199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.8620000000000001,"5min":7.0739999999999998,"15min":7.0739999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.752,"15min":0.89900000000000002},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45600000000000002,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.425,"15min":1.391},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":1.3400000000000001,"15min":1.214},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53100000000000003,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.81000000000000005}]}]},{"osd":0,"up_from":21,"seq":90194313321,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660520,"kb_used_data":241760,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93711320,"statfs":{"total":96636764160,"available":95960391680,"internally_reserved":0,"allocated":247562240,"data_stored":247336352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":428797651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.317},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":7.4400000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246292480,"data_stored":246289001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":185176064,"data_stored":185119807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247361536,"data_stored":247353985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'16433", 'reported_seq': 17154, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.692717+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:06.692717+0000', 'last_peered': '2024-04-22T08:35:06.692717+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:06.692717+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15298560, 'num_objects': 3735, 'num_object_clones': 0, 'num_object_copies': 7470, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16433, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 267}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'16118", 'reported_seq': 16654, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.084817+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:06.084817+0000', 'last_peered': '2024-04-22T08:35:06.084817+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:06.084817+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15339544, 'num_objects': 3746, 'num_object_clones': 0, 'num_object_copies': 7492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3746, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12945, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:12.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'16432", 'reported_seq': 16891, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.190582+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:06.190582+0000', 'last_peered': '2024-04-22T08:35:06.190582+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:06.190582+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15520978, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 7582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16372, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 68}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:12.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:12.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'16363", 'reported_seq': 17246, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.715682+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:06.715682+0000', 'last_peered': '2024-04-22T08:35:06.715682+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:06.715682+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6300", 'ondisk_log_start': "35'6300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15660242, 'num_objects': 3825, 'num_object_clones': 0, 'num_object_copies': 7650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16361, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 454}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:12.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'16187", 'reported_seq': 17024, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.192058+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:06.192058+0000', 'last_peered': '2024-04-22T08:35:06.192058+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:06.192058+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15327232, 'num_objects': 3742, 'num_object_clones': 0, 'num_object_copies': 7484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3410, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12641, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 342, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 332}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:12.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:12.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'16287", 'reported_seq': 17039, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.661647+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:06.661647+0000', 'last_peered': '2024-04-22T08:35:06.661647+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:06.661647+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6200", 'ondisk_log_start': "35'6200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15790697, 'num_objects': 3856, 'num_object_clones': 0, 'num_object_copies': 7712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16286, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:12.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'16062", 'reported_seq': 16733, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.756646+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:06.756646+0000', 'last_peered': '2024-04-22T08:35:06.756646+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:06.756646+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'6000", 'ondisk_log_start': "35'6000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15106048, 'num_objects': 3688, 'num_object_clones': 0, 'num_object_copies': 7376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12293, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 98}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:12.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'16286", 'reported_seq': 16890, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.671394+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:06.671394+0000', 'last_peered': '2024-04-22T08:35:06.671394+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:06.671394+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6200", 'ondisk_log_start': "35'6200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15369426, 'num_objects': 3754, 'num_object_clones': 0, 'num_object_copies': 7508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16284, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:12.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'16308", 'reported_seq': 17024, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:06.709011+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:06.709011+0000', 'last_peered': '2024-04-22T08:35:06.709011+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:06.709011+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6300", 'ondisk_log_start': "35'6300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15133092, 'num_objects': 3698, 'num_object_clones': 0, 'num_object_copies': 7396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16305, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:12.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:12.977 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31463/115466 objects degraded (27.249%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":547,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":57733,"data_bytes":236425069,"bytes_used":1862197248,"bytes_avail":288048095232,"bytes_total":289910292480,"degraded_objects":31463,"degraded_total":115466,"degraded_ratio":0.27248713907124172,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":333,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":456},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:12.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:12.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:13.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:13.577 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:13.624 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-04-22T08:35:12.353407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":236425069,"num_objects":57733,"num_object_clones":0,"num_object_copies":115466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235971,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14144,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160796,"ondisk_log_size":160796,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1818552,"kb_used_data":646620,"kb_used_omap":12,"kb_used_meta":1171891,"kb_avail":281296968,"statfs":{"total":289910292480,"available":288048095232,"internally_reserved":0,"allocated":662138880,"data_stored":661391442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1200016494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,5,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18714624,"num_objects":-4569,"num_object_clones":0,"num_object_copies":-9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006332"},"pg_stats":[{"pgid":"2.f","version":"41'16546","reported_seq":17267,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.587487+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:11.587487+0000","last_peered":"2024-04-22T08:35:11.587487+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:11.587487+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14835712,"num_objects":3622,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16546,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16535","reported_seq":16656,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.010678+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:11.010678+0000","last_peered":"2024-04-22T08:35:11.010678+0000","last_clean":"2024-04-22T08:35:11.010678+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:11.010678+0000","last_undegraded":"2024-04-22T08:35:11.010678+0000","last_fullsized":"2024-04-22T08:35:11.010678+0000","mapping_epoch":27,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15142912,"num_objects":3697,"num_object_clones":0,"num_object_copies":7394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13591,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16275","reported_seq":16811,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.205962+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:11.205962+0000","last_peered":"2024-04-22T08:35:11.205962+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:11.205962+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14696472,"num_objects":3589,"num_object_clones":0,"num_object_copies":7178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13102,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16497","reported_seq":16533,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.112305+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:11.112305+0000","last_peered":"2024-04-22T08:35:11.112305+0000","last_clean":"2024-04-22T08:35:11.112305+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:11.112305+0000","last_undegraded":"2024-04-22T08:35:11.112305+0000","last_fullsized":"2024-04-22T08:35:11.112305+0000","mapping_epoch":27,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14815232,"num_objects":3617,"num_object_clones":0,"num_object_copies":7234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16497,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16367","reported_seq":16511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.691806+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:11.691806+0000","last_peered":"2024-04-22T08:35:11.691806+0000","last_clean":"2024-04-22T08:35:11.691806+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:11.691806+0000","last_undegraded":"2024-04-22T08:35:11.691806+0000","last_fullsized":"2024-04-22T08:35:11.691806+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13459,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16519","reported_seq":24460,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:10.937295+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:10.937295+0000","last_peered":"2024-04-22T08:35:10.937295+0000","last_clean":"2024-04-22T08:35:10.937295+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:10.937295+0000","last_undegraded":"2024-04-22T08:35:10.937295+0000","last_fullsized":"2024-04-22T08:35:10.937295+0000","mapping_epoch":40,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14450688,"num_objects":3528,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13388,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16237","reported_seq":23979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.008057+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:11.008057+0000","last_peered":"2024-04-22T08:35:11.008057+0000","last_clean":"2024-04-22T08:35:11.008057+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:11.008057+0000","last_undegraded":"2024-04-22T08:35:11.008057+0000","last_fullsized":"2024-04-22T08:35:11.008057+0000","mapping_epoch":36,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14856192,"num_objects":3627,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16229","reported_seq":16379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.707169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:11.707169+0000","last_peered":"2024-04-22T08:35:11.707169+0000","last_clean":"2024-04-22T08:35:11.707169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:11.707169+0000","last_undegraded":"2024-04-22T08:35:11.707169+0000","last_fullsized":"2024-04-22T08:35:11.707169+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16588","reported_seq":17047,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.206689+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:11.206689+0000","last_peered":"2024-04-22T08:35:11.206689+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:11.206689+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14882002,"num_objects":3635,"num_object_clones":0,"num_object_copies":7270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16528,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-88}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16502","reported_seq":17385,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.739534+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:11.739534+0000","last_peered":"2024-04-22T08:35:11.739534+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:11.739534+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15090898,"num_objects":3686,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16500,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16328","reported_seq":17349,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.174489+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:11.174489+0000","last_peered":"2024-04-22T08:35:11.174489+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:11.174489+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14749696,"num_objects":3601,"num_object_clones":0,"num_object_copies":7202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12782,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16352","reported_seq":24033,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.151794+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:11.151794+0000","last_peered":"2024-04-22T08:35:11.151794+0000","last_clean":"2024-04-22T08:35:11.151794+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:11.151794+0000","last_undegraded":"2024-04-22T08:35:11.151794+0000","last_fullsized":"2024-04-22T08:35:11.151794+0000","mapping_epoch":38,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14673106,"num_objects":3584,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16437","reported_seq":17194,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.723986+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:11.723986+0000","last_peered":"2024-04-22T08:35:11.723986+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:11.723986+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176297,"num_objects":3706,"num_object_clones":0,"num_object_copies":7412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16436,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16195","reported_seq":16870,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.743379+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:11.743379+0000","last_peered":"2024-04-22T08:35:11.743379+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:11.743379+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561280,"num_objects":3555,"num_object_clones":0,"num_object_copies":7110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16420","reported_seq":17024,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.712836+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:11.712836+0000","last_peered":"2024-04-22T08:35:11.712836+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:11.712836+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14820562,"num_objects":3620,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16418,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16469","reported_seq":17185,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.642185+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:11.642185+0000","last_peered":"2024-04-22T08:35:11.642185+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:11.642185+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14473636,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16466,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-61}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":236425069,"num_objects":57733,"num_object_clones":0,"num_object_copies":115466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235971,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14144,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":659832832,"data_stored":659765545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160796,"ondisk_log_size":160796,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117094,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584108,"kb_used_data":237472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93787732,"statfs":{"total":96636764160,"available":96038637568,"internally_reserved":0,"allocated":243171328,"data_stored":242934906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.379},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":2.2589999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588928,"kb_used_data":174200,"kb_used_omap":4,"kb_used_meta":414715,"kb_avail":93782912,"statfs":{"total":96636764160,"available":96033701888,"internally_reserved":0,"allocated":178380800,"data_stored":178095672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":424668876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.8900000000000001},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":4.0949999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313322,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645516,"kb_used_data":234948,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93726324,"statfs":{"total":96636764160,"available":95975755776,"internally_reserved":0,"allocated":240586752,"data_stored":240360864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239341568,"data_stored":239338089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177848320,"data_stored":177792063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242642944,"data_stored":242635393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:16.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:17.423 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:17.470 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-04-22T08:35:16.359926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231882605,"num_objects":56624,"num_object_clones":0,"num_object_copies":113248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1805128,"kb_used_data":635628,"kb_used_omap":12,"kb_used_meta":1169459,"kb_avail":281310392,"statfs":{"total":289910292480,"available":288061841408,"internally_reserved":0,"allocated":650883072,"data_stored":650135634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1197526126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,7,5,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23257088,"num_objects":-5678,"num_object_clones":0,"num_object_copies":-11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011293"},"pg_stats":[{"pgid":"2.f","version":"41'16546","reported_seq":17267,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.587487+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:11.587487+0000","last_peered":"2024-04-22T08:35:11.587487+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:11.587487+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14835712,"num_objects":3622,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16546,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16655","reported_seq":16776,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.092094+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:16.092094+0000","last_peered":"2024-04-22T08:35:16.092094+0000","last_clean":"2024-04-22T08:35:16.092094+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:16.092094+0000","last_undegraded":"2024-04-22T08:35:16.092094+0000","last_fullsized":"2024-04-22T08:35:16.092094+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13711,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16405","reported_seq":16941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.179721+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:16.179721+0000","last_peered":"2024-04-22T08:35:16.179721+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:16.179721+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14163992,"num_objects":3459,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13232,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-93}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16645","reported_seq":16681,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.219895+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:16.219895+0000","last_peered":"2024-04-22T08:35:16.219895+0000","last_clean":"2024-04-22T08:35:16.219895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:16.219895+0000","last_undegraded":"2024-04-22T08:35:16.219895+0000","last_fullsized":"2024-04-22T08:35:16.219895+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14209024,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16645,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16367","reported_seq":16511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.691806+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:11.691806+0000","last_peered":"2024-04-22T08:35:11.691806+0000","last_clean":"2024-04-22T08:35:11.691806+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:11.691806+0000","last_undegraded":"2024-04-22T08:35:11.691806+0000","last_fullsized":"2024-04-22T08:35:11.691806+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13459,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16666","reported_seq":24608,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.081147+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:16.081147+0000","last_peered":"2024-04-22T08:35:16.081147+0000","last_clean":"2024-04-22T08:35:16.081147+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:16.081147+0000","last_undegraded":"2024-04-22T08:35:16.081147+0000","last_fullsized":"2024-04-22T08:35:16.081147+0000","mapping_epoch":40,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13848576,"num_objects":3381,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16385","reported_seq":24127,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.118525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:16.118525+0000","last_peered":"2024-04-22T08:35:16.118525+0000","last_clean":"2024-04-22T08:35:16.118525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:16.118525+0000","last_undegraded":"2024-04-22T08:35:16.118525+0000","last_fullsized":"2024-04-22T08:35:16.118525+0000","mapping_epoch":36,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16229","reported_seq":16379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.707169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:11.707169+0000","last_peered":"2024-04-22T08:35:11.707169+0000","last_clean":"2024-04-22T08:35:11.707169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:11.707169+0000","last_undegraded":"2024-04-22T08:35:11.707169+0000","last_fullsized":"2024-04-22T08:35:11.707169+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16739","reported_seq":17198,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.177150+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:16.177150+0000","last_peered":"2024-04-22T08:35:16.177150+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:16.177150+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14263506,"num_objects":3484,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16502","reported_seq":17385,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.739534+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:11.739534+0000","last_peered":"2024-04-22T08:35:11.739534+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:11.739534+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15090898,"num_objects":3686,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16500,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16464","reported_seq":17695,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.198474+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:16.198474+0000","last_peered":"2024-04-22T08:35:16.198474+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:16.198474+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14192640,"num_objects":3465,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12918,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16481","reported_seq":24162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.224033+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:16.224033+0000","last_peered":"2024-04-22T08:35:16.224033+0000","last_clean":"2024-04-22T08:35:16.224033+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:16.224033+0000","last_undegraded":"2024-04-22T08:35:16.224033+0000","last_fullsized":"2024-04-22T08:35:16.224033+0000","mapping_epoch":38,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14144722,"num_objects":3455,"num_object_clones":0,"num_object_copies":6910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16479,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16437","reported_seq":17194,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.723986+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:11.723986+0000","last_peered":"2024-04-22T08:35:11.723986+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:11.723986+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176297,"num_objects":3706,"num_object_clones":0,"num_object_copies":7412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16436,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16195","reported_seq":16870,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.743379+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:11.743379+0000","last_peered":"2024-04-22T08:35:11.743379+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:11.743379+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561280,"num_objects":3555,"num_object_clones":0,"num_object_copies":7110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16420","reported_seq":17024,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.712836+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:11.712836+0000","last_peered":"2024-04-22T08:35:11.712836+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:11.712836+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14820562,"num_objects":3620,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16418,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16469","reported_seq":17185,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.642185+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:11.642185+0000","last_peered":"2024-04-22T08:35:11.642185+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:11.642185+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14473636,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16466,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-61}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":231882605,"num_objects":56624,"num_object_clones":0,"num_object_copies":113248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":648527872,"data_stored":648460585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567520,"kb_used_data":233176,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93804320,"statfs":{"total":96636764160,"available":96055623680,"internally_reserved":0,"allocated":238772224,"data_stored":238535802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":1.7629999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.456},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":5.9450000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592092,"kb_used_data":167504,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93779748,"statfs":{"total":96636764160,"available":96030461952,"internally_reserved":0,"allocated":171524096,"data_stored":171238968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":434761420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.253},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313322,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645516,"kb_used_data":234948,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93726324,"statfs":{"total":96636764160,"available":95975755776,"internally_reserved":0,"allocated":240586752,"data_stored":240360864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239341568,"data_stored":239338089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170885120,"data_stored":170828863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238301184,"data_stored":238293633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:17.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:18.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:18.276 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:18.318 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-04-22T08:35:16.359926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231882605,"num_objects":56624,"num_object_clones":0,"num_object_copies":113248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1805128,"kb_used_data":635628,"kb_used_omap":12,"kb_used_meta":1169459,"kb_avail":281310392,"statfs":{"total":289910292480,"available":288061841408,"internally_reserved":0,"allocated":650883072,"data_stored":650135634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1197526126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,7,5,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23257088,"num_objects":-5678,"num_object_clones":0,"num_object_copies":-11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011293"},"pg_stats":[{"pgid":"2.f","version":"41'16546","reported_seq":17267,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.587487+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:11.587487+0000","last_peered":"2024-04-22T08:35:11.587487+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:11.587487+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14835712,"num_objects":3622,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16546,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16655","reported_seq":16776,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.092094+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:16.092094+0000","last_peered":"2024-04-22T08:35:16.092094+0000","last_clean":"2024-04-22T08:35:16.092094+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:16.092094+0000","last_undegraded":"2024-04-22T08:35:16.092094+0000","last_fullsized":"2024-04-22T08:35:16.092094+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13711,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16405","reported_seq":16941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.179721+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:16.179721+0000","last_peered":"2024-04-22T08:35:16.179721+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:16.179721+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14163992,"num_objects":3459,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13232,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-93}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16645","reported_seq":16681,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.219895+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:16.219895+0000","last_peered":"2024-04-22T08:35:16.219895+0000","last_clean":"2024-04-22T08:35:16.219895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:16.219895+0000","last_undegraded":"2024-04-22T08:35:16.219895+0000","last_fullsized":"2024-04-22T08:35:16.219895+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14209024,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16645,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16367","reported_seq":16511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.691806+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:11.691806+0000","last_peered":"2024-04-22T08:35:11.691806+0000","last_clean":"2024-04-22T08:35:11.691806+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:11.691806+0000","last_undegraded":"2024-04-22T08:35:11.691806+0000","last_fullsized":"2024-04-22T08:35:11.691806+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13459,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16666","reported_seq":24608,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.081147+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:16.081147+0000","last_peered":"2024-04-22T08:35:16.081147+0000","last_clean":"2024-04-22T08:35:16.081147+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:16.081147+0000","last_undegraded":"2024-04-22T08:35:16.081147+0000","last_fullsized":"2024-04-22T08:35:16.081147+0000","mapping_epoch":40,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13848576,"num_objects":3381,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16385","reported_seq":24127,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.118525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:16.118525+0000","last_peered":"2024-04-22T08:35:16.118525+0000","last_clean":"2024-04-22T08:35:16.118525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:16.118525+0000","last_undegraded":"2024-04-22T08:35:16.118525+0000","last_fullsized":"2024-04-22T08:35:16.118525+0000","mapping_epoch":36,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16229","reported_seq":16379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:11.707169+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:11.707169+0000","last_peered":"2024-04-22T08:35:11.707169+0000","last_clean":"2024-04-22T08:35:11.707169+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:11.707169+0000","last_undegraded":"2024-04-22T08:35:11.707169+0000","last_fullsized":"2024-04-22T08:35:11.707169+0000","mapping_epoch":27,"log_start":"35'6200","ondisk_log_start":"35'6200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16739","reported_seq":17198,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.177150+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:16.177150+0000","last_peered":"2024-04-22T08:35:16.177150+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:16.177150+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14263506,"num_objects":3484,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16502","reported_seq":17385,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.739534+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:11.739534+0000","last_peered":"2024-04-22T08:35:11.739534+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:11.739534+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15090898,"num_objects":3686,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16500,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16464","reported_seq":17695,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.198474+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:16.198474+0000","last_peered":"2024-04-22T08:35:16.198474+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:16.198474+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14192640,"num_objects":3465,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12918,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16481","reported_seq":24162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.224033+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:16.224033+0000","last_peered":"2024-04-22T08:35:16.224033+0000","last_clean":"2024-04-22T08:35:16.224033+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:16.224033+0000","last_undegraded":"2024-04-22T08:35:16.224033+0000","last_fullsized":"2024-04-22T08:35:16.224033+0000","mapping_epoch":38,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14144722,"num_objects":3455,"num_object_clones":0,"num_object_copies":6910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16479,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16437","reported_seq":17194,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.723986+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:11.723986+0000","last_peered":"2024-04-22T08:35:11.723986+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:11.723986+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176297,"num_objects":3706,"num_object_clones":0,"num_object_copies":7412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16436,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16195","reported_seq":16870,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.743379+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:11.743379+0000","last_peered":"2024-04-22T08:35:11.743379+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:11.743379+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6100","ondisk_log_start":"35'6100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561280,"num_objects":3555,"num_object_clones":0,"num_object_copies":7110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16420","reported_seq":17024,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.712836+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:11.712836+0000","last_peered":"2024-04-22T08:35:11.712836+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:11.712836+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14820562,"num_objects":3620,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16418,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16469","reported_seq":17185,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:11.642185+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:11.642185+0000","last_peered":"2024-04-22T08:35:11.642185+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:11.642185+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14473636,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16466,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-61}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":231882605,"num_objects":56624,"num_object_clones":0,"num_object_copies":113248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":648527872,"data_stored":648460585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567520,"kb_used_data":233176,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93804320,"statfs":{"total":96636764160,"available":96055623680,"internally_reserved":0,"allocated":238772224,"data_stored":238535802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":1.7629999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.456},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":5.9450000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592092,"kb_used_data":167504,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93779748,"statfs":{"total":96636764160,"available":96030461952,"internally_reserved":0,"allocated":171524096,"data_stored":171238968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":434761420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.253},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313322,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645516,"kb_used_data":234948,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93726324,"statfs":{"total":96636764160,"available":95975755776,"internally_reserved":0,"allocated":240586752,"data_stored":240360864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239341568,"data_stored":239338089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170885120,"data_stored":170828863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238301184,"data_stored":238293633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'16546", 'reported_seq': 17267, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.587487+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:11.587487+0000', 'last_peered': '2024-04-22T08:35:11.587487+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:11.587487+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6500", 'ondisk_log_start': "35'6500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14835712, 'num_objects': 3622, 'num_object_clones': 0, 'num_object_copies': 7244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16546, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 154}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'16405", 'reported_seq': 16941, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:16.179721+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:16.179721+0000', 'last_peered': '2024-04-22T08:35:16.179721+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:16.179721+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14163992, 'num_objects': 3459, 'num_object_clones': 0, 'num_object_copies': 6918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3459, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13232, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -93}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'16739", 'reported_seq': 17198, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:16.177150+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:16.177150+0000', 'last_peered': '2024-04-22T08:35:16.177150+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:16.177150+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14263506, 'num_objects': 3484, 'num_object_clones': 0, 'num_object_copies': 6968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16679, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'16502", 'reported_seq': 17385, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.739534+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:11.739534+0000', 'last_peered': '2024-04-22T08:35:11.739534+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:11.739534+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6500", 'ondisk_log_start': "35'6500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15090898, 'num_objects': 3686, 'num_object_clones': 0, 'num_object_copies': 7372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16500, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 315}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.320 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'16464", 'reported_seq': 17695, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:16.198474+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:16.198474+0000', 'last_peered': '2024-04-22T08:35:16.198474+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:16.198474+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14192640, 'num_objects': 3465, 'num_object_clones': 0, 'num_object_copies': 6930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12918, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 539, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 252}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'16437", 'reported_seq': 17194, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.723986+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:11.723986+0000', 'last_peered': '2024-04-22T08:35:11.723986+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:11.723986+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15176297, 'num_objects': 3706, 'num_object_clones': 0, 'num_object_copies': 7412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16436, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'16195", 'reported_seq': 16870, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.743379+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:11.743379+0000', 'last_peered': '2024-04-22T08:35:11.743379+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:11.743379+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'6100", 'ondisk_log_start': "35'6100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14561280, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 7110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12426, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -35}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'16420", 'reported_seq': 17024, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.712836+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:11.712836+0000', 'last_peered': '2024-04-22T08:35:11.712836+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:11.712836+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14820562, 'num_objects': 3620, 'num_object_clones': 0, 'num_object_copies': 7240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16418, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:18.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'16469", 'reported_seq': 17185, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:11.642185+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:11.642185+0000', 'last_peered': '2024-04-22T08:35:11.642185+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:11.642185+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14473636, 'num_objects': 3537, 'num_object_clones': 0, 'num_object_copies': 7074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16466, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -61}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:18.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:19.223 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:19.223 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31358/111008 objects degraded (28.248%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":554,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":55504,"data_bytes":227295085,"bytes_used":1846067200,"bytes_avail":288064225280,"bytes_total":289910292480,"degraded_objects":31358,"degraded_total":111008,"degraded_ratio":0.28248414528682619,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":465},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:19.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:19.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:19.972 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:19.974 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-04-22T08:35:18.367487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227295085,"num_objects":55504,"num_object_clones":0,"num_object_copies":111008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238200,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1802800,"kb_used_data":629196,"kb_used_omap":12,"kb_used_meta":1173555,"kb_avail":281312720,"statfs":{"total":289910292480,"available":288064225280,"internally_reserved":0,"allocated":644296704,"data_stored":643549266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1201720430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,9,6,7,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22925312,"num_objects":-5597,"num_object_clones":0,"num_object_copies":-11194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5597,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.018415"},"pg_stats":[{"pgid":"2.f","version":"41'16702","reported_seq":17423,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.790607+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:16.790607+0000","last_peered":"2024-04-22T08:35:16.790607+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:16.790607+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14196736,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16702,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16655","reported_seq":16776,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.092094+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:16.092094+0000","last_peered":"2024-04-22T08:35:16.092094+0000","last_clean":"2024-04-22T08:35:16.092094+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:16.092094+0000","last_undegraded":"2024-04-22T08:35:16.092094+0000","last_fullsized":"2024-04-22T08:35:16.092094+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651392,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13711,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16405","reported_seq":16941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.179721+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:16.179721+0000","last_peered":"2024-04-22T08:35:16.179721+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:16.179721+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14163992,"num_objects":3459,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13232,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-93}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16645","reported_seq":16681,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.219895+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:16.219895+0000","last_peered":"2024-04-22T08:35:16.219895+0000","last_clean":"2024-04-22T08:35:16.219895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:16.219895+0000","last_undegraded":"2024-04-22T08:35:16.219895+0000","last_fullsized":"2024-04-22T08:35:16.219895+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14209024,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16645,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16485","reported_seq":16629,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.495319+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:16.495319+0000","last_peered":"2024-04-22T08:35:16.495319+0000","last_clean":"2024-04-22T08:35:16.495319+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:16.495319+0000","last_undegraded":"2024-04-22T08:35:16.495319+0000","last_fullsized":"2024-04-22T08:35:16.495319+0000","mapping_epoch":27,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14065664,"num_objects":3434,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13577,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16666","reported_seq":24608,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.081147+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:16.081147+0000","last_peered":"2024-04-22T08:35:16.081147+0000","last_clean":"2024-04-22T08:35:16.081147+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:16.081147+0000","last_undegraded":"2024-04-22T08:35:16.081147+0000","last_fullsized":"2024-04-22T08:35:16.081147+0000","mapping_epoch":40,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13848576,"num_objects":3381,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16385","reported_seq":24127,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.118525+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:16.118525+0000","last_peered":"2024-04-22T08:35:16.118525+0000","last_clean":"2024-04-22T08:35:16.118525+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:16.118525+0000","last_undegraded":"2024-04-22T08:35:16.118525+0000","last_fullsized":"2024-04-22T08:35:16.118525+0000","mapping_epoch":36,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16360","reported_seq":16510,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.699844+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:16.699844+0000","last_peered":"2024-04-22T08:35:16.699844+0000","last_clean":"2024-04-22T08:35:16.699844+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:16.699844+0000","last_undegraded":"2024-04-22T08:35:16.699844+0000","last_fullsized":"2024-04-22T08:35:16.699844+0000","mapping_epoch":27,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14114816,"num_objects":3446,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13008,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16739","reported_seq":17198,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.177150+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:16.177150+0000","last_peered":"2024-04-22T08:35:16.177150+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:16.177150+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14263506,"num_objects":3484,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16640","reported_seq":17524,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.703075+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:16.703075+0000","last_peered":"2024-04-22T08:35:16.703075+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:16.703075+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14525650,"num_objects":3548,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16638,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16464","reported_seq":17695,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.198474+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:16.198474+0000","last_peered":"2024-04-22T08:35:16.198474+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:16.198474+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14192640,"num_objects":3465,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12918,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16481","reported_seq":24162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:16.224033+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:16.224033+0000","last_peered":"2024-04-22T08:35:16.224033+0000","last_clean":"2024-04-22T08:35:16.224033+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:16.224033+0000","last_undegraded":"2024-04-22T08:35:16.224033+0000","last_fullsized":"2024-04-22T08:35:16.224033+0000","mapping_epoch":38,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14144722,"num_objects":3455,"num_object_clones":0,"num_object_copies":6910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16479,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16584","reported_seq":17341,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.790048+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:16.790048+0000","last_peered":"2024-04-22T08:35:16.790048+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:16.790048+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14574185,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16583,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16352","reported_seq":17027,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.779969+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:16.779969+0000","last_peered":"2024-04-22T08:35:16.779969+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:16.779969+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6300","ondisk_log_start":"35'6300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13918208,"num_objects":3398,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12583,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16555","reported_seq":17159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.757493+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:16.757493+0000","last_peered":"2024-04-22T08:35:16.757493+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:16.757493+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16553,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":41}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16607","reported_seq":17323,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:16.819130+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:16.819130+0000","last_peered":"2024-04-22T08:35:16.819130+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:16.819130+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13908388,"num_objects":3399,"num_object_clones":0,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16604,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":227295085,"num_objects":55504,"num_object_clones":0,"num_object_copies":111008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238200,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14249,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":641679360,"data_stored":641612073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567520,"kb_used_data":233176,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93804320,"statfs":{"total":96636764160,"available":96055623680,"internally_reserved":0,"allocated":238772224,"data_stored":238535802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.2789999999999999,"15min":0.94699999999999995},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.0599999999999996,"5min":13.635,"15min":13.635},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.97199999999999998,"15min":0.72199999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.079,"5min":13.721,"15min":13.721},"last":1.7629999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.456},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":5.9450000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592092,"kb_used_data":167504,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93779748,"statfs":{"total":96636764160,"available":96030461952,"internally_reserved":0,"allocated":171524096,"data_stored":171238968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":434761420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.253},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.46200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313324,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643188,"kb_used_data":228516,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93728652,"statfs":{"total":96636764160,"available":95978139648,"internally_reserved":0,"allocated":234000384,"data_stored":233774496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.223},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.255},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232493056,"data_stored":232489577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170885120,"data_stored":170828863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238301184,"data_stored":238293633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:23.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:23.847 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:23.848 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:23.936 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-04-22T08:35:22.368508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1801032,"kb_used_data":611052,"kb_used_omap":12,"kb_used_meta":1189939,"kb_avail":281314488,"statfs":{"total":289910292480,"available":288066035712,"internally_reserved":0,"allocated":625717248,"data_stored":624969810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1218497646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,4,8,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27955200,"num_objects":-6825,"num_object_clones":0,"num_object_copies":-13650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.018453"},"pg_stats":[{"pgid":"2.f","version":"41'16863","reported_seq":17583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.880973+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:21.880973+0000","last_peered":"2024-04-22T08:35:21.880973+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:21.880973+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13537280,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16863,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16799","reported_seq":16920,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.207585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:21.207585+0000","last_peered":"2024-04-22T08:35:21.207585+0000","last_clean":"2024-04-22T08:35:21.207585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:21.207585+0000","last_undegraded":"2024-04-22T08:35:21.207585+0000","last_fullsized":"2024-04-22T08:35:21.207585+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14061568,"num_objects":3433,"num_object_clones":0,"num_object_copies":6866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16552","reported_seq":17088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.191265+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:21.191265+0000","last_peered":"2024-04-22T08:35:21.191265+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:21.191265+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13561880,"num_objects":3312,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13379,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16784","reported_seq":16820,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:20.923668+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:20.923668+0000","last_peered":"2024-04-22T08:35:20.923668+0000","last_clean":"2024-04-22T08:35:20.923668+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:20.923668+0000","last_undegraded":"2024-04-22T08:35:20.923668+0000","last_fullsized":"2024-04-22T08:35:20.923668+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":3330,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16646","reported_seq":16790,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.792630+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:21.792630+0000","last_peered":"2024-04-22T08:35:21.792630+0000","last_clean":"2024-04-22T08:35:21.792630+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:21.792630+0000","last_undegraded":"2024-04-22T08:35:21.792630+0000","last_fullsized":"2024-04-22T08:35:21.792630+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13406208,"num_objects":3273,"num_object_clones":0,"num_object_copies":6546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16801","reported_seq":24742,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.163112+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:21.163112+0000","last_peered":"2024-04-22T08:35:21.163112+0000","last_clean":"2024-04-22T08:35:21.163112+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:21.163112+0000","last_undegraded":"2024-04-22T08:35:21.163112+0000","last_fullsized":"2024-04-22T08:35:21.163112+0000","mapping_epoch":40,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13670,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16527","reported_seq":24269,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.150211+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:21.150211+0000","last_peered":"2024-04-22T08:35:21.150211+0000","last_clean":"2024-04-22T08:35:21.150211+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:21.150211+0000","last_undegraded":"2024-04-22T08:35:21.150211+0000","last_fullsized":"2024-04-22T08:35:21.150211+0000","mapping_epoch":36,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13668352,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16493","reported_seq":16643,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.842580+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:21.842580+0000","last_peered":"2024-04-22T08:35:21.842580+0000","last_clean":"2024-04-22T08:35:21.842580+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:21.842580+0000","last_undegraded":"2024-04-22T08:35:21.842580+0000","last_fullsized":"2024-04-22T08:35:21.842580+0000","mapping_epoch":27,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13570048,"num_objects":3313,"num_object_clones":0,"num_object_copies":6626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16873","reported_seq":17332,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.195538+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:21.195538+0000","last_peered":"2024-04-22T08:35:21.195538+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:21.195538+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13714642,"num_objects":3350,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16813,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16792","reported_seq":17676,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.816371+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:21.816371+0000","last_peered":"2024-04-22T08:35:21.816371+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:21.816371+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13903058,"num_objects":3396,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16790,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16603","reported_seq":18038,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.194158+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:21.194158+0000","last_peered":"2024-04-22T08:35:21.194158+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:21.194158+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13623296,"num_objects":3326,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16613","reported_seq":24292,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.191631+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:21.191631+0000","last_peered":"2024-04-22T08:35:21.191631+0000","last_clean":"2024-04-22T08:35:21.191631+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:21.191631+0000","last_undegraded":"2024-04-22T08:35:21.191631+0000","last_fullsized":"2024-04-22T08:35:21.191631+0000","mapping_epoch":38,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13604050,"num_objects":3323,"num_object_clones":0,"num_object_copies":6646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16611,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16729","reported_seq":17486,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.772903+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:21.772903+0000","last_peered":"2024-04-22T08:35:21.772903+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:21.772903+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980265,"num_objects":3414,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16728,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":29}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16494","reported_seq":17169,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.807862+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:21.807862+0000","last_peered":"2024-04-22T08:35:21.807862+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:21.807862+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13336576,"num_objects":3256,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16703","reported_seq":17308,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.875052+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:21.875052+0000","last_peered":"2024-04-22T08:35:21.875052+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:21.875052+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13661394,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16701,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16751","reported_seq":17467,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.877521+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:21.877521+0000","last_peered":"2024-04-22T08:35:21.877521+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:21.877521+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13318564,"num_objects":3255,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":622809088,"data_stored":622741801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117096,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567140,"kb_used_data":228700,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93804700,"statfs":{"total":96636764160,"available":96056012800,"internally_reserved":0,"allocated":234188800,"data_stored":233952378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":15.6},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.24399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589452,"kb_used_data":160768,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93782388,"statfs":{"total":96636764160,"available":96033165312,"internally_reserved":0,"allocated":164626432,"data_stored":164341304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":438955724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.307},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313325,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644440,"kb_used_data":221584,"kb_used_omap":4,"kb_used_meta":422843,"kb_avail":93727400,"statfs":{"total":96636764160,"available":95976857600,"internally_reserved":0,"allocated":226902016,"data_stored":226676128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":432991955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225275904,"data_stored":225272425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163811328,"data_stored":163755071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233721856,"data_stored":233714305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:23.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:24.747 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-04-22T08:35:24.370975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1801032,"kb_used_data":611052,"kb_used_omap":12,"kb_used_meta":1189939,"kb_avail":281314488,"statfs":{"total":289910292480,"available":288066035712,"internally_reserved":0,"allocated":625717248,"data_stored":624969810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1218497646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,4,8,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18542592,"num_objects":-4527,"num_object_clones":0,"num_object_copies":-9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.017568"},"pg_stats":[{"pgid":"2.f","version":"41'16863","reported_seq":17583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.880973+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:21.880973+0000","last_peered":"2024-04-22T08:35:21.880973+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:21.880973+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13537280,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16863,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16799","reported_seq":16920,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.207585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:21.207585+0000","last_peered":"2024-04-22T08:35:21.207585+0000","last_clean":"2024-04-22T08:35:21.207585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:21.207585+0000","last_undegraded":"2024-04-22T08:35:21.207585+0000","last_fullsized":"2024-04-22T08:35:21.207585+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14061568,"num_objects":3433,"num_object_clones":0,"num_object_copies":6866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16552","reported_seq":17088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.191265+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:21.191265+0000","last_peered":"2024-04-22T08:35:21.191265+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:21.191265+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13561880,"num_objects":3312,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13379,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16784","reported_seq":16820,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:20.923668+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:20.923668+0000","last_peered":"2024-04-22T08:35:20.923668+0000","last_clean":"2024-04-22T08:35:20.923668+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:20.923668+0000","last_undegraded":"2024-04-22T08:35:20.923668+0000","last_fullsized":"2024-04-22T08:35:20.923668+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":3330,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16646","reported_seq":16790,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.792630+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:21.792630+0000","last_peered":"2024-04-22T08:35:21.792630+0000","last_clean":"2024-04-22T08:35:21.792630+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:21.792630+0000","last_undegraded":"2024-04-22T08:35:21.792630+0000","last_fullsized":"2024-04-22T08:35:21.792630+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13406208,"num_objects":3273,"num_object_clones":0,"num_object_copies":6546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16801","reported_seq":24742,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.163112+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:21.163112+0000","last_peered":"2024-04-22T08:35:21.163112+0000","last_clean":"2024-04-22T08:35:21.163112+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:21.163112+0000","last_undegraded":"2024-04-22T08:35:21.163112+0000","last_fullsized":"2024-04-22T08:35:21.163112+0000","mapping_epoch":40,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13670,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16527","reported_seq":24269,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.150211+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:21.150211+0000","last_peered":"2024-04-22T08:35:21.150211+0000","last_clean":"2024-04-22T08:35:21.150211+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:21.150211+0000","last_undegraded":"2024-04-22T08:35:21.150211+0000","last_fullsized":"2024-04-22T08:35:21.150211+0000","mapping_epoch":36,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13668352,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16493","reported_seq":16643,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.842580+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:21.842580+0000","last_peered":"2024-04-22T08:35:21.842580+0000","last_clean":"2024-04-22T08:35:21.842580+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:21.842580+0000","last_undegraded":"2024-04-22T08:35:21.842580+0000","last_fullsized":"2024-04-22T08:35:21.842580+0000","mapping_epoch":27,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13570048,"num_objects":3313,"num_object_clones":0,"num_object_copies":6626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16873","reported_seq":17332,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.195538+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:21.195538+0000","last_peered":"2024-04-22T08:35:21.195538+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:21.195538+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13714642,"num_objects":3350,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16813,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16792","reported_seq":17676,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.816371+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:21.816371+0000","last_peered":"2024-04-22T08:35:21.816371+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:21.816371+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13903058,"num_objects":3396,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16790,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16603","reported_seq":18038,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.194158+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:21.194158+0000","last_peered":"2024-04-22T08:35:21.194158+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:21.194158+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13623296,"num_objects":3326,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16613","reported_seq":24292,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.191631+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:21.191631+0000","last_peered":"2024-04-22T08:35:21.191631+0000","last_clean":"2024-04-22T08:35:21.191631+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:21.191631+0000","last_undegraded":"2024-04-22T08:35:21.191631+0000","last_fullsized":"2024-04-22T08:35:21.191631+0000","mapping_epoch":38,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13604050,"num_objects":3323,"num_object_clones":0,"num_object_copies":6646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16611,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16729","reported_seq":17486,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.772903+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:21.772903+0000","last_peered":"2024-04-22T08:35:21.772903+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:21.772903+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980265,"num_objects":3414,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16728,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":29}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16494","reported_seq":17169,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.807862+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:21.807862+0000","last_peered":"2024-04-22T08:35:21.807862+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:21.807862+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13336576,"num_objects":3256,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16703","reported_seq":17308,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.875052+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:21.875052+0000","last_peered":"2024-04-22T08:35:21.875052+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:21.875052+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13661394,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16701,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16751","reported_seq":17467,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.877521+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:21.877521+0000","last_peered":"2024-04-22T08:35:21.877521+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:21.877521+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13318564,"num_objects":3255,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":622809088,"data_stored":622741801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117096,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567140,"kb_used_data":228700,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93804700,"statfs":{"total":96636764160,"available":96056012800,"internally_reserved":0,"allocated":234188800,"data_stored":233952378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":15.6},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.24399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589452,"kb_used_data":160768,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93782388,"statfs":{"total":96636764160,"available":96033165312,"internally_reserved":0,"allocated":164626432,"data_stored":164341304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":438955724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.307},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313325,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644440,"kb_used_data":221584,"kb_used_omap":4,"kb_used_meta":422843,"kb_avail":93727400,"statfs":{"total":96636764160,"available":95976857600,"internally_reserved":0,"allocated":226902016,"data_stored":226676128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":432991955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225275904,"data_stored":225272425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163811328,"data_stored":163755071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233721856,"data_stored":233714305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:24.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:24.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'16863", 'reported_seq': 17583, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.880973+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:21.880973+0000', 'last_peered': '2024-04-22T08:35:21.880973+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:21.880973+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6800", 'ondisk_log_start': "35'6800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13537280, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 6610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16863, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -163}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'16552", 'reported_seq': 17088, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.191265+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:21.191265+0000', 'last_peered': '2024-04-22T08:35:21.191265+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:21.191265+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'6500", 'ondisk_log_start': "35'6500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13561880, 'num_objects': 3312, 'num_object_clones': 0, 'num_object_copies': 6624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3312, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13379, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'16873", 'reported_seq': 17332, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.195538+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:21.195538+0000', 'last_peered': '2024-04-22T08:35:21.195538+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:21.195538+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'6800", 'ondisk_log_start': "35'6800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13714642, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 6700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16813, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'16792", 'reported_seq': 17676, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.816371+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:21.816371+0000', 'last_peered': '2024-04-22T08:35:21.816371+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:21.816371+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13903058, 'num_objects': 3396, 'num_object_clones': 0, 'num_object_copies': 6792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16790, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 25}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:24.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'16603", 'reported_seq': 18038, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.194158+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:21.194158+0000', 'last_peered': '2024-04-22T08:35:21.194158+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:21.194158+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'6600", 'ondisk_log_start': "35'6600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13623296, 'num_objects': 3326, 'num_object_clones': 0, 'num_object_copies': 6652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3111, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13057, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 641, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 215}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'16729", 'reported_seq': 17486, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.772903+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:21.772903+0000', 'last_peered': '2024-04-22T08:35:21.772903+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:21.772903+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13980265, 'num_objects': 3414, 'num_object_clones': 0, 'num_object_copies': 6828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16728, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 29}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'16494", 'reported_seq': 17169, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.807862+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:21.807862+0000', 'last_peered': '2024-04-22T08:35:21.807862+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:21.807862+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'6400", 'ondisk_log_start': "35'6400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13336576, 'num_objects': 3256, 'num_object_clones': 0, 'num_object_copies': 6512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12725, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'16703", 'reported_seq': 17308, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.875052+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:21.875052+0000', 'last_peered': '2024-04-22T08:35:21.875052+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:21.875052+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13661394, 'num_objects': 3337, 'num_object_clones': 0, 'num_object_copies': 6674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16701, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:24.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'16751", 'reported_seq': 17467, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:21.877521+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:21.877521+0000', 'last_peered': '2024-04-22T08:35:21.877521+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:21.877521+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13318564, 'num_objects': 3255, 'num_object_clones': 0, 'num_object_copies': 6510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16748, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:24.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:25.587 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:25.587 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31256/106412 objects degraded (29.373%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":560,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":53206,"data_bytes":217882477,"bytes_used":1844256768,"bytes_avail":288066035712,"bytes_total":289910292480,"degraded_objects":31256,"degraded_total":106412,"degraded_ratio":0.29372627147314212,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":376},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:25.607 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:25.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:25.891 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:25.892 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:25.911 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-04-22T08:35:24.370975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1801032,"kb_used_data":611052,"kb_used_omap":12,"kb_used_meta":1189939,"kb_avail":281314488,"statfs":{"total":289910292480,"available":288066035712,"internally_reserved":0,"allocated":625717248,"data_stored":624969810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1218497646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,4,8,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18542592,"num_objects":-4527,"num_object_clones":0,"num_object_copies":-9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.017568"},"pg_stats":[{"pgid":"2.f","version":"41'16863","reported_seq":17583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.880973+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:21.880973+0000","last_peered":"2024-04-22T08:35:21.880973+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:21.880973+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13537280,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16863,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16799","reported_seq":16920,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.207585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:21.207585+0000","last_peered":"2024-04-22T08:35:21.207585+0000","last_clean":"2024-04-22T08:35:21.207585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:21.207585+0000","last_undegraded":"2024-04-22T08:35:21.207585+0000","last_fullsized":"2024-04-22T08:35:21.207585+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14061568,"num_objects":3433,"num_object_clones":0,"num_object_copies":6866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16552","reported_seq":17088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.191265+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:21.191265+0000","last_peered":"2024-04-22T08:35:21.191265+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:21.191265+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13561880,"num_objects":3312,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13379,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16784","reported_seq":16820,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:20.923668+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:20.923668+0000","last_peered":"2024-04-22T08:35:20.923668+0000","last_clean":"2024-04-22T08:35:20.923668+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:20.923668+0000","last_undegraded":"2024-04-22T08:35:20.923668+0000","last_fullsized":"2024-04-22T08:35:20.923668+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":3330,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16784,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16646","reported_seq":16790,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.792630+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:21.792630+0000","last_peered":"2024-04-22T08:35:21.792630+0000","last_clean":"2024-04-22T08:35:21.792630+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:21.792630+0000","last_undegraded":"2024-04-22T08:35:21.792630+0000","last_fullsized":"2024-04-22T08:35:21.792630+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13406208,"num_objects":3273,"num_object_clones":0,"num_object_copies":6546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13738,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16801","reported_seq":24742,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.163112+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:21.163112+0000","last_peered":"2024-04-22T08:35:21.163112+0000","last_clean":"2024-04-22T08:35:21.163112+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:21.163112+0000","last_undegraded":"2024-04-22T08:35:21.163112+0000","last_fullsized":"2024-04-22T08:35:21.163112+0000","mapping_epoch":40,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13670,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16527","reported_seq":24269,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.150211+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:21.150211+0000","last_peered":"2024-04-22T08:35:21.150211+0000","last_clean":"2024-04-22T08:35:21.150211+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:21.150211+0000","last_undegraded":"2024-04-22T08:35:21.150211+0000","last_fullsized":"2024-04-22T08:35:21.150211+0000","mapping_epoch":36,"log_start":"35'6500","ondisk_log_start":"35'6500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13668352,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12895,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16493","reported_seq":16643,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.842580+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:21.842580+0000","last_peered":"2024-04-22T08:35:21.842580+0000","last_clean":"2024-04-22T08:35:21.842580+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:21.842580+0000","last_undegraded":"2024-04-22T08:35:21.842580+0000","last_fullsized":"2024-04-22T08:35:21.842580+0000","mapping_epoch":27,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13570048,"num_objects":3313,"num_object_clones":0,"num_object_copies":6626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'16873","reported_seq":17332,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.195538+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:21.195538+0000","last_peered":"2024-04-22T08:35:21.195538+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:21.195538+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13714642,"num_objects":3350,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16813,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581990+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:01.581990+0000","last_peered":"2024-04-22T08:35:01.581990+0000","last_clean":"2024-04-22T08:35:01.581990+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:01.581990+0000","last_undegraded":"2024-04-22T08:35:01.581990+0000","last_fullsized":"2024-04-22T08:35:01.581990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16792","reported_seq":17676,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.816371+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:21.816371+0000","last_peered":"2024-04-22T08:35:21.816371+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:21.816371+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13903058,"num_objects":3396,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16790,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16603","reported_seq":18038,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.194158+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:21.194158+0000","last_peered":"2024-04-22T08:35:21.194158+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:21.194158+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13623296,"num_objects":3326,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16613","reported_seq":24292,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:21.191631+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:21.191631+0000","last_peered":"2024-04-22T08:35:21.191631+0000","last_clean":"2024-04-22T08:35:21.191631+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:21.191631+0000","last_undegraded":"2024-04-22T08:35:21.191631+0000","last_fullsized":"2024-04-22T08:35:21.191631+0000","mapping_epoch":38,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13604050,"num_objects":3323,"num_object_clones":0,"num_object_copies":6646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16611,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16729","reported_seq":17486,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.772903+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:21.772903+0000","last_peered":"2024-04-22T08:35:21.772903+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:21.772903+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980265,"num_objects":3414,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16728,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":29}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16494","reported_seq":17169,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.807862+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:21.807862+0000","last_peered":"2024-04-22T08:35:21.807862+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:21.807862+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6400","ondisk_log_start":"35'6400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13336576,"num_objects":3256,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583221+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:01.583221+0000","last_peered":"2024-04-22T08:35:01.583221+0000","last_clean":"2024-04-22T08:35:01.583221+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:01.583221+0000","last_undegraded":"2024-04-22T08:35:01.583221+0000","last_fullsized":"2024-04-22T08:35:01.583221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16703","reported_seq":17308,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.875052+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:21.875052+0000","last_peered":"2024-04-22T08:35:21.875052+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:21.875052+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13661394,"num_objects":3337,"num_object_clones":0,"num_object_copies":6674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16701,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16751","reported_seq":17467,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:21.877521+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:21.877521+0000","last_peered":"2024-04-22T08:35:21.877521+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:21.877521+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13318564,"num_objects":3255,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217882477,"num_objects":53206,"num_object_clones":0,"num_object_copies":106412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240498,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14351,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":622809088,"data_stored":622741801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117096,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567140,"kb_used_data":228700,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93804700,"statfs":{"total":96636764160,"available":96056012800,"internally_reserved":0,"allocated":234188800,"data_stored":233952378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":15.6},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.24399999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589452,"kb_used_data":160768,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93782388,"statfs":{"total":96636764160,"available":96033165312,"internally_reserved":0,"allocated":164626432,"data_stored":164341304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":438955724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.307},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313325,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644440,"kb_used_data":221584,"kb_used_omap":4,"kb_used_meta":422843,"kb_avail":93727400,"statfs":{"total":96636764160,"available":95976857600,"internally_reserved":0,"allocated":226902016,"data_stored":226676128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":432991955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.20200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.23,"15min":0.88},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":4.4210000000000003,"5min":8,"15min":8},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.877,"5min":0.997,"15min":0.86499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.2919999999999998,"5min":12.131,"15min":12.131},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225275904,"data_stored":225272425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163811328,"data_stored":163755071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233721856,"data_stored":233714305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:28.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:29.799 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:29.892 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-04-22T08:35:28.375482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":208924525,"num_objects":51019,"num_object_clones":0,"num_object_copies":102038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242685,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14478,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1801636,"kb_used_data":593608,"kb_used_omap":12,"kb_used_meta":1207987,"kb_avail":281313884,"statfs":{"total":289910292480,"available":288065417216,"internally_reserved":0,"allocated":607854592,"data_stored":607107154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1236978798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,8,7,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22958080,"num_objects":-5605,"num_object_clones":0,"num_object_copies":-11210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015556"},"pg_stats":[{"pgid":"2.f","version":"41'16991","reported_seq":17712,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.820611+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:26.820611+0000","last_peered":"2024-04-22T08:35:26.820611+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:26.820611+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13012992,"num_objects":3177,"num_object_clones":0,"num_object_copies":6354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16991,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-291}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16936","reported_seq":17057,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.151434+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:26.151434+0000","last_peered":"2024-04-22T08:35:26.151434+0000","last_clean":"2024-04-22T08:35:26.151434+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:26.151434+0000","last_undegraded":"2024-04-22T08:35:26.151434+0000","last_fullsized":"2024-04-22T08:35:26.151434+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":3296,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13992,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16693","reported_seq":17227,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.207554+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:26.207554+0000","last_peered":"2024-04-22T08:35:26.207554+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:26.207554+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12984344,"num_objects":3171,"num_object_clones":0,"num_object_copies":6342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13520,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16906","reported_seq":16942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.226744+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:26.226744+0000","last_peered":"2024-04-22T08:35:26.226744+0000","last_clean":"2024-04-22T08:35:26.226744+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:26.226744+0000","last_undegraded":"2024-04-22T08:35:26.226744+0000","last_fullsized":"2024-04-22T08:35:26.226744+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13139968,"num_objects":3208,"num_object_clones":0,"num_object_copies":6416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16906,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16816","reported_seq":16960,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.860480+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:26.860480+0000","last_peered":"2024-04-22T08:35:26.860480+0000","last_clean":"2024-04-22T08:35:26.860480+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:26.860480+0000","last_undegraded":"2024-04-22T08:35:26.860480+0000","last_fullsized":"2024-04-22T08:35:26.860480+0000","mapping_epoch":27,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12709888,"num_objects":3103,"num_object_clones":0,"num_object_copies":6206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13908,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16926","reported_seq":24867,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.231040+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:26.231040+0000","last_peered":"2024-04-22T08:35:26.231040+0000","last_clean":"2024-04-22T08:35:26.231040+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:26.231040+0000","last_undegraded":"2024-04-22T08:35:26.231040+0000","last_fullsized":"2024-04-22T08:35:26.231040+0000","mapping_epoch":40,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12783616,"num_objects":3121,"num_object_clones":0,"num_object_copies":6242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13795,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16663","reported_seq":24405,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.151852+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:26.151852+0000","last_peered":"2024-04-22T08:35:26.151852+0000","last_clean":"2024-04-22T08:35:26.151852+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:26.151852+0000","last_undegraded":"2024-04-22T08:35:26.151852+0000","last_fullsized":"2024-04-22T08:35:26.151852+0000","mapping_epoch":36,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13111296,"num_objects":3201,"num_object_clones":0,"num_object_copies":6402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16638","reported_seq":16786,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.846271+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:26.846271+0000","last_peered":"2024-04-22T08:35:26.846271+0000","last_clean":"2024-04-22T08:35:26.846271+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:26.846271+0000","last_undegraded":"2024-04-22T08:35:26.846271+0000","last_fullsized":"2024-04-22T08:35:26.846271+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13286,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17019","reported_seq":17478,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.191798+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:26.191798+0000","last_peered":"2024-04-22T08:35:26.191798+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:26.191798+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13116626,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16959,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16917","reported_seq":17801,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.819660+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:26.819660+0000","last_peered":"2024-04-22T08:35:26.819660+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:26.819660+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13391058,"num_objects":3271,"num_object_clones":0,"num_object_copies":6542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16915,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16732","reported_seq":18420,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.263959+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:26.263959+0000","last_peered":"2024-04-22T08:35:26.263959+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:26.263959+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13094912,"num_objects":3197,"num_object_clones":0,"num_object_copies":6394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16749","reported_seq":24430,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.228065+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:26.228065+0000","last_peered":"2024-04-22T08:35:26.228065+0000","last_clean":"2024-04-22T08:35:26.228065+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:26.228065+0000","last_undegraded":"2024-04-22T08:35:26.228065+0000","last_fullsized":"2024-04-22T08:35:26.228065+0000","mapping_epoch":38,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046994,"num_objects":3187,"num_object_clones":0,"num_object_copies":6374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16747,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16863","reported_seq":17620,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.848054+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:26.848054+0000","last_peered":"2024-04-22T08:35:26.848054+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:26.848054+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13431401,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16862,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16622","reported_seq":17297,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.875310+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:26.875310+0000","last_peered":"2024-04-22T08:35:26.875310+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:26.875310+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812288,"num_objects":3128,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12853,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16850","reported_seq":17454,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.787724+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:26.787724+0000","last_peered":"2024-04-22T08:35:26.787724+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:26.787724+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059282,"num_objects":3190,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16848,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16889","reported_seq":17605,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.829015+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:26.829015+0000","last_peered":"2024-04-22T08:35:26.829015+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:26.829015+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12753316,"num_objects":3117,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16886,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":208924525,"num_objects":51019,"num_object_clones":0,"num_object_copies":102038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242685,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14478,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604913664,"data_stored":604846377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117097,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566728,"kb_used_data":224188,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93805112,"statfs":{"total":96636764160,"available":96056434688,"internally_reserved":0,"allocated":229568512,"data_stored":229332090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.223},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.435}]}]},{"osd":1,"up_from":33,"seq":141733920859,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609188,"kb_used_data":154268,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93762652,"statfs":{"total":96636764160,"available":96012955648,"internally_reserved":0,"allocated":157970432,"data_stored":157685304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.217}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.40500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313326,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625720,"kb_used_data":215152,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93746120,"statfs":{"total":96636764160,"available":95996026880,"internally_reserved":0,"allocated":220315648,"data_stored":220089760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.2489999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":6.9160000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218624000,"data_stored":218620521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157151232,"data_stored":157094975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229138432,"data_stored":229130881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:29.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:30.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:30.691 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:30.741 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-04-22T08:35:30.375925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":208924525,"num_objects":51019,"num_object_clones":0,"num_object_copies":102038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242685,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14478,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1801636,"kb_used_data":593608,"kb_used_omap":12,"kb_used_meta":1207987,"kb_avail":281313884,"statfs":{"total":289910292480,"available":288065417216,"internally_reserved":0,"allocated":607854592,"data_stored":607107154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1236978798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,8,7,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18370560,"num_objects":-4485,"num_object_clones":0,"num_object_copies":-8970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4485,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008438"},"pg_stats":[{"pgid":"2.f","version":"41'16991","reported_seq":17712,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.820611+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:26.820611+0000","last_peered":"2024-04-22T08:35:26.820611+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:26.820611+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13012992,"num_objects":3177,"num_object_clones":0,"num_object_copies":6354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16991,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-291}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'16936","reported_seq":17057,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.151434+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:26.151434+0000","last_peered":"2024-04-22T08:35:26.151434+0000","last_clean":"2024-04-22T08:35:26.151434+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:26.151434+0000","last_undegraded":"2024-04-22T08:35:26.151434+0000","last_fullsized":"2024-04-22T08:35:26.151434+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":3296,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13992,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16693","reported_seq":17227,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.207554+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:26.207554+0000","last_peered":"2024-04-22T08:35:26.207554+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:26.207554+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12984344,"num_objects":3171,"num_object_clones":0,"num_object_copies":6342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13520,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'16906","reported_seq":16942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.226744+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:26.226744+0000","last_peered":"2024-04-22T08:35:26.226744+0000","last_clean":"2024-04-22T08:35:26.226744+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:26.226744+0000","last_undegraded":"2024-04-22T08:35:26.226744+0000","last_fullsized":"2024-04-22T08:35:26.226744+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13139968,"num_objects":3208,"num_object_clones":0,"num_object_copies":6416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16906,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16816","reported_seq":16960,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.860480+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:26.860480+0000","last_peered":"2024-04-22T08:35:26.860480+0000","last_clean":"2024-04-22T08:35:26.860480+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:26.860480+0000","last_undegraded":"2024-04-22T08:35:26.860480+0000","last_fullsized":"2024-04-22T08:35:26.860480+0000","mapping_epoch":27,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12709888,"num_objects":3103,"num_object_clones":0,"num_object_copies":6206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13908,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'16926","reported_seq":24867,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.231040+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:26.231040+0000","last_peered":"2024-04-22T08:35:26.231040+0000","last_clean":"2024-04-22T08:35:26.231040+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:26.231040+0000","last_undegraded":"2024-04-22T08:35:26.231040+0000","last_fullsized":"2024-04-22T08:35:26.231040+0000","mapping_epoch":40,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12783616,"num_objects":3121,"num_object_clones":0,"num_object_copies":6242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13795,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16663","reported_seq":24405,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.151852+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:26.151852+0000","last_peered":"2024-04-22T08:35:26.151852+0000","last_clean":"2024-04-22T08:35:26.151852+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:26.151852+0000","last_undegraded":"2024-04-22T08:35:26.151852+0000","last_fullsized":"2024-04-22T08:35:26.151852+0000","mapping_epoch":36,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13111296,"num_objects":3201,"num_object_clones":0,"num_object_copies":6402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16638","reported_seq":16786,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.846271+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:26.846271+0000","last_peered":"2024-04-22T08:35:26.846271+0000","last_clean":"2024-04-22T08:35:26.846271+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:26.846271+0000","last_undegraded":"2024-04-22T08:35:26.846271+0000","last_fullsized":"2024-04-22T08:35:26.846271+0000","mapping_epoch":27,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13286,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17019","reported_seq":17478,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.191798+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:26.191798+0000","last_peered":"2024-04-22T08:35:26.191798+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:26.191798+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13116626,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16959,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'16917","reported_seq":17801,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.819660+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:26.819660+0000","last_peered":"2024-04-22T08:35:26.819660+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:26.819660+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13391058,"num_objects":3271,"num_object_clones":0,"num_object_copies":6542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16915,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16732","reported_seq":18420,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.263959+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:26.263959+0000","last_peered":"2024-04-22T08:35:26.263959+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:26.263959+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13094912,"num_objects":3197,"num_object_clones":0,"num_object_copies":6394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16749","reported_seq":24430,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:26.228065+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:26.228065+0000","last_peered":"2024-04-22T08:35:26.228065+0000","last_clean":"2024-04-22T08:35:26.228065+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:26.228065+0000","last_undegraded":"2024-04-22T08:35:26.228065+0000","last_fullsized":"2024-04-22T08:35:26.228065+0000","mapping_epoch":38,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13046994,"num_objects":3187,"num_object_clones":0,"num_object_copies":6374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16747,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'16863","reported_seq":17620,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.848054+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:26.848054+0000","last_peered":"2024-04-22T08:35:26.848054+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:26.848054+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13431401,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16862,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16622","reported_seq":17297,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.875310+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:26.875310+0000","last_peered":"2024-04-22T08:35:26.875310+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:26.875310+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6600","ondisk_log_start":"35'6600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812288,"num_objects":3128,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12853,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'16850","reported_seq":17454,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.787724+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:26.787724+0000","last_peered":"2024-04-22T08:35:26.787724+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:26.787724+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059282,"num_objects":3190,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16848,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'16889","reported_seq":17605,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:26.829015+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:26.829015+0000","last_peered":"2024-04-22T08:35:26.829015+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:26.829015+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12753316,"num_objects":3117,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16886,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":208924525,"num_objects":51019,"num_object_clones":0,"num_object_copies":102038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242685,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14478,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604913664,"data_stored":604846377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117097,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566728,"kb_used_data":224188,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93805112,"statfs":{"total":96636764160,"available":96056434688,"internally_reserved":0,"allocated":229568512,"data_stored":229332090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.223},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.435}]}]},{"osd":1,"up_from":33,"seq":141733920859,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609188,"kb_used_data":154268,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93762652,"statfs":{"total":96636764160,"available":96012955648,"internally_reserved":0,"allocated":157970432,"data_stored":157685304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.217}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.40500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313326,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625720,"kb_used_data":215152,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93746120,"statfs":{"total":96636764160,"available":95996026880,"internally_reserved":0,"allocated":220315648,"data_stored":220089760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":5.2489999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":6.9160000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218624000,"data_stored":218620521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157151232,"data_stored":157094975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229138432,"data_stored":229130881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:30.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:30.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'16991", 'reported_seq': 17712, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.820611+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:26.820611+0000', 'last_peered': '2024-04-22T08:35:26.820611+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:26.820611+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'6900", 'ondisk_log_start': "35'6900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13012992, 'num_objects': 3177, 'num_object_clones': 0, 'num_object_copies': 6354, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16991, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -291}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.742 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'16693", 'reported_seq': 17227, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.207554+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:26.207554+0000', 'last_peered': '2024-04-22T08:35:26.207554+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:26.207554+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'6600", 'ondisk_log_start': "35'6600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12984344, 'num_objects': 3171, 'num_object_clones': 0, 'num_object_copies': 6342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3171, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13520, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'17019", 'reported_seq': 17478, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.191798+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:26.191798+0000', 'last_peered': '2024-04-22T08:35:26.191798+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:26.191798+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13116626, 'num_objects': 3204, 'num_object_clones': 0, 'num_object_copies': 6408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16959, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'16917", 'reported_seq': 17801, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.819660+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:26.819660+0000', 'last_peered': '2024-04-22T08:35:26.819660+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:26.819660+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'6900", 'ondisk_log_start': "35'6900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13391058, 'num_objects': 3271, 'num_object_clones': 0, 'num_object_copies': 6542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16915, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'16732", 'reported_seq': 18420, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.263959+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:26.263959+0000', 'last_peered': '2024-04-22T08:35:26.263959+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:26.263959+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13094912, 'num_objects': 3197, 'num_object_clones': 0, 'num_object_copies': 6394, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2984, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13186, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 768, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 213}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'16863", 'reported_seq': 17620, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.848054+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:26.848054+0000', 'last_peered': '2024-04-22T08:35:26.848054+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:26.848054+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'6800", 'ondisk_log_start': "35'6800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13431401, 'num_objects': 3280, 'num_object_clones': 0, 'num_object_copies': 6560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16862, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:30.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'16622", 'reported_seq': 17297, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.875310+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:26.875310+0000', 'last_peered': '2024-04-22T08:35:26.875310+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:26.875310+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'6600", 'ondisk_log_start': "35'6600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812288, 'num_objects': 3128, 'num_object_clones': 0, 'num_object_copies': 6256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12853, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:30.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'16850", 'reported_seq': 17454, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.787724+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:26.787724+0000', 'last_peered': '2024-04-22T08:35:26.787724+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:26.787724+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6800", 'ondisk_log_start': "35'6800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13059282, 'num_objects': 3190, 'num_object_clones': 0, 'num_object_copies': 6380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16848, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:30.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'16889", 'reported_seq': 17605, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:26.829015+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:26.829015+0000', 'last_peered': '2024-04-22T08:35:26.829015+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:26.829015+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'6800", 'ondisk_log_start': "35'6800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12753316, 'num_objects': 3117, 'num_object_clones': 0, 'num_object_copies': 6234, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16886, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:30.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:31.688 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:31.688 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31129/102038 objects degraded (30.507%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":566,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":51019,"data_bytes":208924525,"bytes_used":1844875264,"bytes_avail":288065417216,"bytes_total":289910292480,"degraded_objects":31129,"degraded_total":102038,"degraded_ratio":0.30507262000431212,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":229,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":373},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:31.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:31.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:32.618 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:32.619 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-04-22T08:35:32.376584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199323501,"num_objects":48675,"num_object_clones":0,"num_object_copies":97350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245029,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14580,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160954,"ondisk_log_size":160954,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1782928,"kb_used_data":574896,"kb_used_omap":12,"kb_used_meta":1207987,"kb_avail":281332592,"statfs":{"total":289910292480,"available":288084574208,"internally_reserved":0,"allocated":588693504,"data_stored":587946066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1236978798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,2,9,15,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27971584,"num_objects":-6829,"num_object_clones":0,"num_object_copies":-13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008531"},"pg_stats":[{"pgid":"2.f","version":"41'17139","reported_seq":17860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.848271+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:31.848271+0000","last_peered":"2024-04-22T08:35:31.848271+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:31.848271+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12406784,"num_objects":3029,"num_object_clones":0,"num_object_copies":6058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17090","reported_seq":17211,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.270346+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:31.270346+0000","last_peered":"2024-04-22T08:35:31.270346+0000","last_clean":"2024-04-22T08:35:31.270346+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:31.270346+0000","last_undegraded":"2024-04-22T08:35:31.270346+0000","last_fullsized":"2024-04-22T08:35:31.270346+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12869632,"num_objects":3142,"num_object_clones":0,"num_object_copies":6284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14146,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16842","reported_seq":17378,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.059701+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:31.059701+0000","last_peered":"2024-04-22T08:35:31.059701+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:31.059701+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12374040,"num_objects":3022,"num_object_clones":0,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13669,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17031","reported_seq":17067,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.219013+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:31.219013+0000","last_peered":"2024-04-22T08:35:31.219013+0000","last_clean":"2024-04-22T08:35:31.219013+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:31.219013+0000","last_undegraded":"2024-04-22T08:35:31.219013+0000","last_fullsized":"2024-04-22T08:35:31.219013+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12627968,"num_objects":3083,"num_object_clones":0,"num_object_copies":6166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17031,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16945","reported_seq":17089,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.823752+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:31.823752+0000","last_peered":"2024-04-22T08:35:31.823752+0000","last_clean":"2024-04-22T08:35:31.823752+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:31.823752+0000","last_undegraded":"2024-04-22T08:35:31.823752+0000","last_fullsized":"2024-04-22T08:35:31.823752+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":2974,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14037,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17056","reported_seq":24998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.201086+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:31.201086+0000","last_peered":"2024-04-22T08:35:31.201086+0000","last_clean":"2024-04-22T08:35:31.201086+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:31.201086+0000","last_undegraded":"2024-04-22T08:35:31.201086+0000","last_fullsized":"2024-04-22T08:35:31.201086+0000","mapping_epoch":40,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12251136,"num_objects":2991,"num_object_clones":0,"num_object_copies":5982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13925,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16796","reported_seq":24538,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.184467+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:31.184467+0000","last_peered":"2024-04-22T08:35:31.184467+0000","last_clean":"2024-04-22T08:35:31.184467+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:31.184467+0000","last_undegraded":"2024-04-22T08:35:31.184467+0000","last_fullsized":"2024-04-22T08:35:31.184467+0000","mapping_epoch":36,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12566528,"num_objects":3068,"num_object_clones":0,"num_object_copies":6136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16784","reported_seq":16934,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.919440+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:31.919440+0000","last_peered":"2024-04-22T08:35:31.919440+0000","last_clean":"2024-04-22T08:35:31.919440+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:31.919440+0000","last_undegraded":"2024-04-22T08:35:31.919440+0000","last_fullsized":"2024-04-22T08:35:31.919440+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12378112,"num_objects":3022,"num_object_clones":0,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13432,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17160","reported_seq":17619,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.219510+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:31.219510+0000","last_peered":"2024-04-22T08:35:31.219510+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:31.219510+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12539090,"num_objects":3063,"num_object_clones":0,"num_object_copies":6126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17100,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-660}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17079","reported_seq":17963,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.914581+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:31.914581+0000","last_peered":"2024-04-22T08:35:31.914581+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:31.914581+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12727506,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17077,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'16879","reported_seq":18771,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.274045+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:31.274045+0000","last_peered":"2024-04-22T08:35:31.274045+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:31.274045+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'6800","ondisk_log_start":"35'6800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12492800,"num_objects":3050,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13333,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'16921","reported_seq":24602,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.207796+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:31.207796+0000","last_peered":"2024-04-22T08:35:31.207796+0000","last_clean":"2024-04-22T08:35:31.207796+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:31.207796+0000","last_undegraded":"2024-04-22T08:35:31.207796+0000","last_fullsized":"2024-04-22T08:35:31.207796+0000","mapping_epoch":38,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12342482,"num_objects":3015,"num_object_clones":0,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16919,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17009","reported_seq":17766,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.884373+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:31.884373+0000","last_peered":"2024-04-22T08:35:31.884373+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:31.884373+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12833385,"num_objects":3134,"num_object_clones":0,"num_object_copies":6268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17008,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16774","reported_seq":17449,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.881838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:31.881838+0000","last_peered":"2024-04-22T08:35:31.881838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:31.881838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":2976,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13005,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17000","reported_seq":17605,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.886423+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:31.886423+0000","last_peered":"2024-04-22T08:35:31.886423+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:31.886423+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16998,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17049","reported_seq":17765,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.851602+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:31.851602+0000","last_peered":"2024-04-22T08:35:31.851602+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:31.851602+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12097956,"num_objects":2957,"num_object_clones":0,"num_object_copies":5914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17046,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":199323501,"num_objects":48675,"num_object_clones":0,"num_object_copies":97350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245029,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14580,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585576448,"data_stored":585509161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160954,"ondisk_log_size":160954,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117098,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566180,"kb_used_data":219548,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93805660,"statfs":{"total":96636764160,"available":96056995840,"internally_reserved":0,"allocated":224817152,"data_stored":224580730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":2.1850000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":2.2130000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.312},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.28399999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920860,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593908,"kb_used_data":147180,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93777932,"statfs":{"total":96636764160,"available":96028602368,"internally_reserved":0,"allocated":150712320,"data_stored":150427192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":457436876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.9960000000000004},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":5.0309999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.26700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313327,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622840,"kb_used_data":208168,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93749000,"statfs":{"total":96636764160,"available":95998976000,"internally_reserved":0,"allocated":213164032,"data_stored":212938144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,9,11,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":7.4390000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.774},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211513344,"data_stored":211509865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149745664,"data_stored":149689407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224317440,"data_stored":224309889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:35.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:36.452 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:36.453 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:36.524 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-04-22T08:35:36.381117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194428781,"num_objects":47480,"num_object_clones":0,"num_object_copies":94960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246224,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1766916,"kb_used_data":562964,"kb_used_omap":12,"kb_used_meta":1203891,"kb_avail":281348604,"statfs":{"total":289910292480,"available":288100970496,"internally_reserved":0,"allocated":576475136,"data_stored":575727698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1232784494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,10,17,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23453696,"num_objects":-5726,"num_object_clones":0,"num_object_copies":-11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010141"},"pg_stats":[{"pgid":"2.f","version":"41'17139","reported_seq":17860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.848271+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:31.848271+0000","last_peered":"2024-04-22T08:35:31.848271+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:31.848271+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12406784,"num_objects":3029,"num_object_clones":0,"num_object_copies":6058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17229","reported_seq":17350,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.273717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:36.273717+0000","last_peered":"2024-04-22T08:35:36.273717+0000","last_clean":"2024-04-22T08:35:36.273717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:36.273717+0000","last_undegraded":"2024-04-22T08:35:36.273717+0000","last_fullsized":"2024-04-22T08:35:36.273717+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12300288,"num_objects":3003,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14285,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16999","reported_seq":17535,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.256675+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:36.256675+0000","last_peered":"2024-04-22T08:35:36.256675+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:36.256675+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730968,"num_objects":2865,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13826,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17178","reported_seq":17214,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.129500+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:36.129500+0000","last_peered":"2024-04-22T08:35:36.129500+0000","last_clean":"2024-04-22T08:35:36.129500+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:36.129500+0000","last_undegraded":"2024-04-22T08:35:36.129500+0000","last_fullsized":"2024-04-22T08:35:36.129500+0000","mapping_epoch":27,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12025856,"num_objects":2936,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17178,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16945","reported_seq":17089,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.823752+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:31.823752+0000","last_peered":"2024-04-22T08:35:31.823752+0000","last_clean":"2024-04-22T08:35:31.823752+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:31.823752+0000","last_undegraded":"2024-04-22T08:35:31.823752+0000","last_fullsized":"2024-04-22T08:35:31.823752+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":2974,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14037,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17220","reported_seq":25162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.314749+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:36.314749+0000","last_peered":"2024-04-22T08:35:36.314749+0000","last_clean":"2024-04-22T08:35:36.314749+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:36.314749+0000","last_undegraded":"2024-04-22T08:35:36.314749+0000","last_fullsized":"2024-04-22T08:35:36.314749+0000","mapping_epoch":40,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11579392,"num_objects":2827,"num_object_clones":0,"num_object_copies":5654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16933","reported_seq":24675,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.307767+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:36.307767+0000","last_peered":"2024-04-22T08:35:36.307767+0000","last_clean":"2024-04-22T08:35:36.307767+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:36.307767+0000","last_undegraded":"2024-04-22T08:35:36.307767+0000","last_fullsized":"2024-04-22T08:35:36.307767+0000","mapping_epoch":36,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12005376,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13301,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16784","reported_seq":16934,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.919440+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:31.919440+0000","last_peered":"2024-04-22T08:35:31.919440+0000","last_clean":"2024-04-22T08:35:31.919440+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:31.919440+0000","last_undegraded":"2024-04-22T08:35:31.919440+0000","last_fullsized":"2024-04-22T08:35:31.919440+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12378112,"num_objects":3022,"num_object_clones":0,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13432,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17326","reported_seq":17784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.323454+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:36.323454+0000","last_peered":"2024-04-22T08:35:36.323454+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:36.323454+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11859154,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17266,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17079","reported_seq":17963,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.914581+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:31.914581+0000","last_peered":"2024-04-22T08:35:31.914581+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:31.914581+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12727506,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17077,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17029","reported_seq":19134,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.318263+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:36.318263+0000","last_peered":"2024-04-22T08:35:36.318263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:36.318263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11878400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17056","reported_seq":24737,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.288466+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:36.288466+0000","last_peered":"2024-04-22T08:35:36.288466+0000","last_clean":"2024-04-22T08:35:36.288466+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:36.288466+0000","last_undegraded":"2024-04-22T08:35:36.288466+0000","last_fullsized":"2024-04-22T08:35:36.288466+0000","mapping_epoch":38,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11789522,"num_objects":2880,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17054,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17009","reported_seq":17766,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.884373+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:31.884373+0000","last_peered":"2024-04-22T08:35:31.884373+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:31.884373+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12833385,"num_objects":3134,"num_object_clones":0,"num_object_copies":6268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17008,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16774","reported_seq":17449,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.881838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:31.881838+0000","last_peered":"2024-04-22T08:35:31.881838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:31.881838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":2976,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13005,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17000","reported_seq":17605,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.886423+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:31.886423+0000","last_peered":"2024-04-22T08:35:31.886423+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:31.886423+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16998,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17049","reported_seq":17765,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.851602+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:31.851602+0000","last_peered":"2024-04-22T08:35:31.851602+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:31.851602+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12097956,"num_objects":2957,"num_object_clones":0,"num_object_copies":5914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17046,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":194428781,"num_objects":47480,"num_object_clones":0,"num_object_copies":94960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246224,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573296640,"data_stored":573229353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117099,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549200,"kb_used_data":214856,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93822640,"statfs":{"total":96636764160,"available":96074383360,"internally_reserved":0,"allocated":220012544,"data_stored":219776122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.42099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920861,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594876,"kb_used_data":139940,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93776964,"statfs":{"total":96636764160,"available":96027611136,"internally_reserved":0,"allocated":143298560,"data_stored":143013432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":3.7090000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313327,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622840,"kb_used_data":208168,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93749000,"statfs":{"total":96636764160,"available":95998976000,"internally_reserved":0,"allocated":213164032,"data_stored":212938144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,9,11,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":7.4390000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.774},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211513344,"data_stored":211509865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142331904,"data_stored":142275647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219451392,"data_stored":219443841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:36.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:37.315 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:37.382 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-04-22T08:35:36.381117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194428781,"num_objects":47480,"num_object_clones":0,"num_object_copies":94960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246224,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1766916,"kb_used_data":562964,"kb_used_omap":12,"kb_used_meta":1203891,"kb_avail":281348604,"statfs":{"total":289910292480,"available":288100970496,"internally_reserved":0,"allocated":576475136,"data_stored":575727698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1232784494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,10,17,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23453696,"num_objects":-5726,"num_object_clones":0,"num_object_copies":-11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010141"},"pg_stats":[{"pgid":"2.f","version":"41'17139","reported_seq":17860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.848271+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:31.848271+0000","last_peered":"2024-04-22T08:35:31.848271+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:31.848271+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12406784,"num_objects":3029,"num_object_clones":0,"num_object_copies":6058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17229","reported_seq":17350,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.273717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:36.273717+0000","last_peered":"2024-04-22T08:35:36.273717+0000","last_clean":"2024-04-22T08:35:36.273717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:36.273717+0000","last_undegraded":"2024-04-22T08:35:36.273717+0000","last_fullsized":"2024-04-22T08:35:36.273717+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12300288,"num_objects":3003,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14285,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16999","reported_seq":17535,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.256675+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:36.256675+0000","last_peered":"2024-04-22T08:35:36.256675+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:36.256675+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730968,"num_objects":2865,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13826,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17178","reported_seq":17214,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.129500+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:36.129500+0000","last_peered":"2024-04-22T08:35:36.129500+0000","last_clean":"2024-04-22T08:35:36.129500+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:36.129500+0000","last_undegraded":"2024-04-22T08:35:36.129500+0000","last_fullsized":"2024-04-22T08:35:36.129500+0000","mapping_epoch":27,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12025856,"num_objects":2936,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17178,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'16945","reported_seq":17089,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.823752+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:31.823752+0000","last_peered":"2024-04-22T08:35:31.823752+0000","last_clean":"2024-04-22T08:35:31.823752+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:31.823752+0000","last_undegraded":"2024-04-22T08:35:31.823752+0000","last_fullsized":"2024-04-22T08:35:31.823752+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":2974,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14037,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17220","reported_seq":25162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.314749+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:36.314749+0000","last_peered":"2024-04-22T08:35:36.314749+0000","last_clean":"2024-04-22T08:35:36.314749+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:36.314749+0000","last_undegraded":"2024-04-22T08:35:36.314749+0000","last_fullsized":"2024-04-22T08:35:36.314749+0000","mapping_epoch":40,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11579392,"num_objects":2827,"num_object_clones":0,"num_object_copies":5654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16933","reported_seq":24675,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.307767+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:36.307767+0000","last_peered":"2024-04-22T08:35:36.307767+0000","last_clean":"2024-04-22T08:35:36.307767+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:36.307767+0000","last_undegraded":"2024-04-22T08:35:36.307767+0000","last_fullsized":"2024-04-22T08:35:36.307767+0000","mapping_epoch":36,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12005376,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13301,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16784","reported_seq":16934,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:31.919440+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:31.919440+0000","last_peered":"2024-04-22T08:35:31.919440+0000","last_clean":"2024-04-22T08:35:31.919440+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:31.919440+0000","last_undegraded":"2024-04-22T08:35:31.919440+0000","last_fullsized":"2024-04-22T08:35:31.919440+0000","mapping_epoch":27,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12378112,"num_objects":3022,"num_object_clones":0,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13432,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17326","reported_seq":17784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.323454+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:36.323454+0000","last_peered":"2024-04-22T08:35:36.323454+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:36.323454+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11859154,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17266,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17079","reported_seq":17963,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.914581+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:31.914581+0000","last_peered":"2024-04-22T08:35:31.914581+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:31.914581+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12727506,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17077,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17029","reported_seq":19134,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.318263+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:36.318263+0000","last_peered":"2024-04-22T08:35:36.318263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:36.318263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11878400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17056","reported_seq":24737,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.288466+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:36.288466+0000","last_peered":"2024-04-22T08:35:36.288466+0000","last_clean":"2024-04-22T08:35:36.288466+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:36.288466+0000","last_undegraded":"2024-04-22T08:35:36.288466+0000","last_fullsized":"2024-04-22T08:35:36.288466+0000","mapping_epoch":38,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11789522,"num_objects":2880,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17054,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17009","reported_seq":17766,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.884373+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:31.884373+0000","last_peered":"2024-04-22T08:35:31.884373+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:31.884373+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12833385,"num_objects":3134,"num_object_clones":0,"num_object_copies":6268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17008,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16774","reported_seq":17449,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.881838+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:31.881838+0000","last_peered":"2024-04-22T08:35:31.881838+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:31.881838+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6700","ondisk_log_start":"35'6700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":2976,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13005,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17000","reported_seq":17605,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.886423+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:31.886423+0000","last_peered":"2024-04-22T08:35:31.886423+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:31.886423+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16998,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17049","reported_seq":17765,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:31.851602+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:31.851602+0000","last_peered":"2024-04-22T08:35:31.851602+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:31.851602+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12097956,"num_objects":2957,"num_object_clones":0,"num_object_copies":5914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17046,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":194428781,"num_objects":47480,"num_object_clones":0,"num_object_copies":94960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246224,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573296640,"data_stored":573229353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117099,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549200,"kb_used_data":214856,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93822640,"statfs":{"total":96636764160,"available":96074383360,"internally_reserved":0,"allocated":220012544,"data_stored":219776122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.42099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920861,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594876,"kb_used_data":139940,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93776964,"statfs":{"total":96636764160,"available":96027611136,"internally_reserved":0,"allocated":143298560,"data_stored":143013432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":3.7090000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313327,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622840,"kb_used_data":208168,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93749000,"statfs":{"total":96636764160,"available":95998976000,"internally_reserved":0,"allocated":213164032,"data_stored":212938144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,9,11,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":7.4390000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.774},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211513344,"data_stored":211509865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142331904,"data_stored":142275647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219451392,"data_stored":219443841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'17139", 'reported_seq': 17860, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.848271+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:31.848271+0000', 'last_peered': '2024-04-22T08:35:31.848271+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:31.848271+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'7100", 'ondisk_log_start': "35'7100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12406784, 'num_objects': 3029, 'num_object_clones': 0, 'num_object_copies': 6058, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17139, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -439}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'16999", 'reported_seq': 17535, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:36.256675+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:36.256675+0000', 'last_peered': '2024-04-22T08:35:36.256675+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:36.256675+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'6900", 'ondisk_log_start': "35'6900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11730968, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 5730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13826, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'17326", 'reported_seq': 17784, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:36.323454+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:36.323454+0000', 'last_peered': '2024-04-22T08:35:36.323454+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:36.323454+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11859154, 'num_objects': 2897, 'num_object_clones': 0, 'num_object_copies': 5794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17266, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:37.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'17079", 'reported_seq': 17963, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.914581+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:31.914581+0000', 'last_peered': '2024-04-22T08:35:31.914581+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:31.914581+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12727506, 'num_objects': 3109, 'num_object_clones': 0, 'num_object_copies': 6218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17077, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -262}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17029", 'reported_seq': 19134, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:36.318263+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:36.318263+0000', 'last_peered': '2024-04-22T08:35:36.318263+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:36.318263+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11878400, 'num_objects': 2900, 'num_object_clones': 0, 'num_object_copies': 5800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2776, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13483, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 976, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 124}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'17009", 'reported_seq': 17766, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.884373+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:31.884373+0000', 'last_peered': '2024-04-22T08:35:31.884373+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:31.884373+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12833385, 'num_objects': 3134, 'num_object_clones': 0, 'num_object_copies': 6268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17008, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'16774", 'reported_seq': 17449, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.881838+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:31.881838+0000', 'last_peered': '2024-04-22T08:35:31.881838+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:31.881838+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'6700", 'ondisk_log_start': "35'6700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12189696, 'num_objects': 2976, 'num_object_clones': 0, 'num_object_copies': 5952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13005, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'17000", 'reported_seq': 17605, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.886423+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:31.886423+0000', 'last_peered': '2024-04-22T08:35:31.886423+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:31.886423+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'6900", 'ondisk_log_start': "35'6900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12444882, 'num_objects': 3040, 'num_object_clones': 0, 'num_object_copies': 6080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16998, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:37.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'17049", 'reported_seq': 17765, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:31.851602+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:31.851602+0000', 'last_peered': '2024-04-22T08:35:31.851602+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:31.851602+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12097956, 'num_objects': 2957, 'num_object_clones': 0, 'num_object_copies': 5914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17046, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:37.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:38.136 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:38.136 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30921/94960 objects degraded (32.562%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":47480,"data_bytes":194428781,"bytes_used":1809321984,"bytes_avail":288100970496,"bytes_total":289910292480,"degraded_objects":30921,"degraded_total":94960,"degraded_ratio":0.32562131423757373,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":335,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":476},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:38.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:38.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:38.995 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:38.996 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:39.052 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-22T08:35:38.381783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189517677,"num_objects":46281,"num_object_clones":0,"num_object_copies":92562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247423,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1751620,"kb_used_data":555852,"kb_used_omap":12,"kb_used_meta":1195699,"kb_avail":281363900,"statfs":{"total":289910292480,"available":288116633600,"internally_reserved":0,"allocated":569192448,"data_stored":568445010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1224395886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,1,5,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23973888,"num_objects":-5853,"num_object_clones":0,"num_object_copies":-11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006793"},"pg_stats":[{"pgid":"2.f","version":"41'17282","reported_seq":18003,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.895611+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:36.895611+0000","last_peered":"2024-04-22T08:35:36.895611+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:36.895611+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11821056,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17282,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17229","reported_seq":17350,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.273717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:36.273717+0000","last_peered":"2024-04-22T08:35:36.273717+0000","last_clean":"2024-04-22T08:35:36.273717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:36.273717+0000","last_undegraded":"2024-04-22T08:35:36.273717+0000","last_fullsized":"2024-04-22T08:35:36.273717+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12300288,"num_objects":3003,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14285,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'16999","reported_seq":17535,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.256675+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:36.256675+0000","last_peered":"2024-04-22T08:35:36.256675+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:36.256675+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730968,"num_objects":2865,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13826,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17178","reported_seq":17214,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.129500+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:36.129500+0000","last_peered":"2024-04-22T08:35:36.129500+0000","last_clean":"2024-04-22T08:35:36.129500+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:36.129500+0000","last_undegraded":"2024-04-22T08:35:36.129500+0000","last_fullsized":"2024-04-22T08:35:36.129500+0000","mapping_epoch":27,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12025856,"num_objects":2936,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17178,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17088","reported_seq":17232,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.950767+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:36.950767+0000","last_peered":"2024-04-22T08:35:36.950767+0000","last_clean":"2024-04-22T08:35:36.950767+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:36.950767+0000","last_undegraded":"2024-04-22T08:35:36.950767+0000","last_fullsized":"2024-04-22T08:35:36.950767+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14180,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17220","reported_seq":25162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.314749+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:36.314749+0000","last_peered":"2024-04-22T08:35:36.314749+0000","last_clean":"2024-04-22T08:35:36.314749+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:36.314749+0000","last_undegraded":"2024-04-22T08:35:36.314749+0000","last_fullsized":"2024-04-22T08:35:36.314749+0000","mapping_epoch":40,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11579392,"num_objects":2827,"num_object_clones":0,"num_object_copies":5654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'16933","reported_seq":24675,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.307767+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:36.307767+0000","last_peered":"2024-04-22T08:35:36.307767+0000","last_clean":"2024-04-22T08:35:36.307767+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:36.307767+0000","last_undegraded":"2024-04-22T08:35:36.307767+0000","last_fullsized":"2024-04-22T08:35:36.307767+0000","mapping_epoch":36,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12005376,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13301,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'16945","reported_seq":17095,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.955095+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:36.955095+0000","last_peered":"2024-04-22T08:35:36.955095+0000","last_clean":"2024-04-22T08:35:36.955095+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:36.955095+0000","last_undegraded":"2024-04-22T08:35:36.955095+0000","last_fullsized":"2024-04-22T08:35:36.955095+0000","mapping_epoch":27,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11718656,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13593,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17326","reported_seq":17784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.323454+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:36.323454+0000","last_peered":"2024-04-22T08:35:36.323454+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:36.323454+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11859154,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17266,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17241","reported_seq":18125,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.965433+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:36.965433+0000","last_peered":"2024-04-22T08:35:36.965433+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:36.965433+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063954,"num_objects":2947,"num_object_clones":0,"num_object_copies":5894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17239,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17029","reported_seq":19134,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.318263+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:36.318263+0000","last_peered":"2024-04-22T08:35:36.318263+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:36.318263+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11878400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13483,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17056","reported_seq":24737,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:36.288466+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:36.288466+0000","last_peered":"2024-04-22T08:35:36.288466+0000","last_clean":"2024-04-22T08:35:36.288466+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:36.288466+0000","last_undegraded":"2024-04-22T08:35:36.288466+0000","last_fullsized":"2024-04-22T08:35:36.288466+0000","mapping_epoch":38,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11789522,"num_objects":2880,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17054,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17181","reported_seq":17938,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.893699+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:36.893699+0000","last_peered":"2024-04-22T08:35:36.893699+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:36.893699+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12128873,"num_objects":2962,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17180,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'16925","reported_seq":17599,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.923471+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:36.923471+0000","last_peered":"2024-04-22T08:35:36.923471+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:36.923471+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'6900","ondisk_log_start":"35'6900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11571200,"num_objects":2825,"num_object_clones":0,"num_object_copies":5650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13156,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17130","reported_seq":17735,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.853775+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:36.853775+0000","last_peered":"2024-04-22T08:35:36.853775+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:36.853775+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11912402,"num_objects":2910,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17128,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17186","reported_seq":17902,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:36.962172+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:36.962172+0000","last_peered":"2024-04-22T08:35:36.962172+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:36.962172+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11536804,"num_objects":2820,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17183,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":189517677,"num_objects":46281,"num_object_clones":0,"num_object_copies":92562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247423,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14686,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565927936,"data_stored":565860649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117099,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549200,"kb_used_data":214856,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93822640,"statfs":{"total":96636764160,"available":96074383360,"internally_reserved":0,"allocated":220012544,"data_stored":219776122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.42099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920861,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594876,"kb_used_data":139940,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93776964,"statfs":{"total":96636764160,"available":96027611136,"internally_reserved":0,"allocated":143298560,"data_stored":143013432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":3.7090000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.32400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313328,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607544,"kb_used_data":201056,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93764296,"statfs":{"total":96636764160,"available":96014639104,"internally_reserved":0,"allocated":205881344,"data_stored":205655456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":416214739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204144640,"data_stored":204141161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142331904,"data_stored":142275647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219451392,"data_stored":219443841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:42.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:42.884 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:42.885 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:42.965 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-22T08:35:42.382676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1731860,"kb_used_data":536124,"kb_used_omap":12,"kb_used_meta":1195699,"kb_avail":281383660,"statfs":{"total":289910292480,"available":288136867840,"internally_reserved":0,"allocated":548990976,"data_stored":548243538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1224395886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,8,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29003776,"num_objects":-7081,"num_object_clones":0,"num_object_copies":-14162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006750"},"pg_stats":[{"pgid":"2.f","version":"41'17427","reported_seq":18148,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:42.013261+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:42.013261+0000","last_peered":"2024-04-22T08:35:42.013261+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:42.013261+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227136,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17390","reported_seq":17511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.319275+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:41.319275+0000","last_peered":"2024-04-22T08:35:41.319275+0000","last_clean":"2024-04-22T08:35:41.319275+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:41.319275+0000","last_undegraded":"2024-04-22T08:35:41.319275+0000","last_fullsized":"2024-04-22T08:35:41.319275+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":2842,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14446,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17142","reported_seq":17678,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.166148+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:41.166148+0000","last_peered":"2024-04-22T08:35:41.166148+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:41.166148+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11145240,"num_objects":2722,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13969,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17322","reported_seq":17357,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.350641+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:41.350641+0000","last_peered":"2024-04-22T08:35:41.350641+0000","last_clean":"2024-04-22T08:35:41.350641+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:41.350641+0000","last_undegraded":"2024-04-22T08:35:41.350641+0000","last_fullsized":"2024-04-22T08:35:41.350641+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11436032,"num_objects":2792,"num_object_clones":0,"num_object_copies":5584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17322,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17232","reported_seq":17377,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:42.012352+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:42.012352+0000","last_peered":"2024-04-22T08:35:42.012352+0000","last_clean":"2024-04-22T08:35:42.012352+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:42.012352+0000","last_undegraded":"2024-04-22T08:35:42.012352+0000","last_fullsized":"2024-04-22T08:35:42.012352+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11005952,"num_objects":2687,"num_object_clones":0,"num_object_copies":5374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14324,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17381","reported_seq":25322,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.361221+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:41.361221+0000","last_peered":"2024-04-22T08:35:41.361221+0000","last_clean":"2024-04-22T08:35:41.361221+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:41.361221+0000","last_undegraded":"2024-04-22T08:35:41.361221+0000","last_fullsized":"2024-04-22T08:35:41.361221+0000","mapping_epoch":40,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":2666,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14250,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17085","reported_seq":24827,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.373194+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:41.373194+0000","last_peered":"2024-04-22T08:35:41.373194+0000","last_clean":"2024-04-22T08:35:41.373194+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:41.373194+0000","last_undegraded":"2024-04-22T08:35:41.373194+0000","last_fullsized":"2024-04-22T08:35:41.373194+0000","mapping_epoch":36,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17092","reported_seq":17243,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.911638+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:41.911638+0000","last_peered":"2024-04-22T08:35:41.911638+0000","last_clean":"2024-04-22T08:35:41.911638+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:41.911638+0000","last_undegraded":"2024-04-22T08:35:41.911638+0000","last_fullsized":"2024-04-22T08:35:41.911638+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":2714,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13740,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17471","reported_seq":17930,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.306625+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:41.306625+0000","last_peered":"2024-04-22T08:35:41.306625+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:41.306625+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11265234,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17411,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17381","reported_seq":18265,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.943770+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:41.943770+0000","last_peered":"2024-04-22T08:35:41.943770+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:41.943770+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11490514,"num_objects":2807,"num_object_clones":0,"num_object_copies":5614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17379,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17176","reported_seq":19459,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.300435+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:41.300435+0000","last_peered":"2024-04-22T08:35:41.300435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:41.300435+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":66}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17188","reported_seq":24869,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.326377+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:41.326377+0000","last_peered":"2024-04-22T08:35:41.326377+0000","last_clean":"2024-04-22T08:35:41.326377+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:41.326377+0000","last_undegraded":"2024-04-22T08:35:41.326377+0000","last_fullsized":"2024-04-22T08:35:41.326377+0000","mapping_epoch":38,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248850,"num_objects":2748,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17186,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17335","reported_seq":18090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.999533+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:41.999533+0000","last_peered":"2024-04-22T08:35:41.999533+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:41.999533+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498089,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17334,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17052","reported_seq":17727,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.913132+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:41.913132+0000","last_peered":"2024-04-22T08:35:41.913132+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:41.913132+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11051008,"num_objects":2698,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13283,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17295","reported_seq":17898,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.948635+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:41.948635+0000","last_peered":"2024-04-22T08:35:41.948635+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:41.948635+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236562,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17293,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17322","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.984670+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:41.984670+0000","last_peered":"2024-04-22T08:35:41.984670+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:41.984670+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10979748,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17319,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546803712,"data_stored":546736425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548828,"kb_used_data":210388,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93823012,"statfs":{"total":96636764160,"available":96074764288,"internally_reserved":0,"allocated":215437312,"data_stored":215200890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":2.0339999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575676,"kb_used_data":133044,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93796164,"statfs":{"total":96636764160,"available":96047271936,"internally_reserved":0,"allocated":136237056,"data_stored":135951928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":453242572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.8929999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313329,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607356,"kb_used_data":192692,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93764484,"statfs":{"total":96636764160,"available":96014831616,"internally_reserved":0,"allocated":197316608,"data_stored":197090720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.249},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":196902912,"data_stored":196899433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135180288,"data_stored":135124031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214720512,"data_stored":214712961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:42.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:43.655 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:43.745 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-22T08:35:42.382676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1731860,"kb_used_data":536124,"kb_used_omap":12,"kb_used_meta":1195699,"kb_avail":281383660,"statfs":{"total":289910292480,"available":288136867840,"internally_reserved":0,"allocated":548990976,"data_stored":548243538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1224395886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,8,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29003776,"num_objects":-7081,"num_object_clones":0,"num_object_copies":-14162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006750"},"pg_stats":[{"pgid":"2.f","version":"41'17427","reported_seq":18148,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:42.013261+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:42.013261+0000","last_peered":"2024-04-22T08:35:42.013261+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:42.013261+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227136,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17390","reported_seq":17511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.319275+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:41.319275+0000","last_peered":"2024-04-22T08:35:41.319275+0000","last_clean":"2024-04-22T08:35:41.319275+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:41.319275+0000","last_undegraded":"2024-04-22T08:35:41.319275+0000","last_fullsized":"2024-04-22T08:35:41.319275+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":2842,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14446,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17142","reported_seq":17678,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.166148+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:41.166148+0000","last_peered":"2024-04-22T08:35:41.166148+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:41.166148+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11145240,"num_objects":2722,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13969,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17322","reported_seq":17357,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.350641+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:41.350641+0000","last_peered":"2024-04-22T08:35:41.350641+0000","last_clean":"2024-04-22T08:35:41.350641+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:41.350641+0000","last_undegraded":"2024-04-22T08:35:41.350641+0000","last_fullsized":"2024-04-22T08:35:41.350641+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11436032,"num_objects":2792,"num_object_clones":0,"num_object_copies":5584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17322,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17232","reported_seq":17377,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:42.012352+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:42.012352+0000","last_peered":"2024-04-22T08:35:42.012352+0000","last_clean":"2024-04-22T08:35:42.012352+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:42.012352+0000","last_undegraded":"2024-04-22T08:35:42.012352+0000","last_fullsized":"2024-04-22T08:35:42.012352+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11005952,"num_objects":2687,"num_object_clones":0,"num_object_copies":5374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14324,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17381","reported_seq":25322,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.361221+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:41.361221+0000","last_peered":"2024-04-22T08:35:41.361221+0000","last_clean":"2024-04-22T08:35:41.361221+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:41.361221+0000","last_undegraded":"2024-04-22T08:35:41.361221+0000","last_fullsized":"2024-04-22T08:35:41.361221+0000","mapping_epoch":40,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":2666,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14250,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17085","reported_seq":24827,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.373194+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:41.373194+0000","last_peered":"2024-04-22T08:35:41.373194+0000","last_clean":"2024-04-22T08:35:41.373194+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:41.373194+0000","last_undegraded":"2024-04-22T08:35:41.373194+0000","last_fullsized":"2024-04-22T08:35:41.373194+0000","mapping_epoch":36,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17092","reported_seq":17243,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.911638+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:41.911638+0000","last_peered":"2024-04-22T08:35:41.911638+0000","last_clean":"2024-04-22T08:35:41.911638+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:41.911638+0000","last_undegraded":"2024-04-22T08:35:41.911638+0000","last_fullsized":"2024-04-22T08:35:41.911638+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":2714,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13740,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17471","reported_seq":17930,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.306625+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:41.306625+0000","last_peered":"2024-04-22T08:35:41.306625+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:41.306625+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11265234,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17411,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17381","reported_seq":18265,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.943770+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:41.943770+0000","last_peered":"2024-04-22T08:35:41.943770+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:41.943770+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11490514,"num_objects":2807,"num_object_clones":0,"num_object_copies":5614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17379,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17176","reported_seq":19459,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.300435+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:41.300435+0000","last_peered":"2024-04-22T08:35:41.300435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:41.300435+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":66}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17188","reported_seq":24869,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.326377+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:41.326377+0000","last_peered":"2024-04-22T08:35:41.326377+0000","last_clean":"2024-04-22T08:35:41.326377+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:41.326377+0000","last_undegraded":"2024-04-22T08:35:41.326377+0000","last_fullsized":"2024-04-22T08:35:41.326377+0000","mapping_epoch":38,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248850,"num_objects":2748,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17186,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17335","reported_seq":18090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.999533+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:41.999533+0000","last_peered":"2024-04-22T08:35:41.999533+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:41.999533+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498089,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17334,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17052","reported_seq":17727,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.913132+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:41.913132+0000","last_peered":"2024-04-22T08:35:41.913132+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:41.913132+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11051008,"num_objects":2698,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13283,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17295","reported_seq":17898,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.948635+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:41.948635+0000","last_peered":"2024-04-22T08:35:41.948635+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:41.948635+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236562,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17293,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17322","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.984670+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:41.984670+0000","last_peered":"2024-04-22T08:35:41.984670+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:41.984670+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10979748,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17319,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546803712,"data_stored":546736425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548828,"kb_used_data":210388,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93823012,"statfs":{"total":96636764160,"available":96074764288,"internally_reserved":0,"allocated":215437312,"data_stored":215200890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":2.0339999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575676,"kb_used_data":133044,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93796164,"statfs":{"total":96636764160,"available":96047271936,"internally_reserved":0,"allocated":136237056,"data_stored":135951928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":453242572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.8929999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313329,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607356,"kb_used_data":192692,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93764484,"statfs":{"total":96636764160,"available":96014831616,"internally_reserved":0,"allocated":197316608,"data_stored":197090720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.249},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":196902912,"data_stored":196899433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135180288,"data_stored":135124031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214720512,"data_stored":214712961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'17427", 'reported_seq': 18148, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:42.013261+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:42.013261+0000', 'last_peered': '2024-04-22T08:35:42.013261+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:42.013261+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11227136, 'num_objects': 2741, 'num_object_clones': 0, 'num_object_copies': 5482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17427, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -727}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17142", 'reported_seq': 17678, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.166148+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:41.166148+0000', 'last_peered': '2024-04-22T08:35:41.166148+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:41.166148+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7100", 'ondisk_log_start': "35'7100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11145240, 'num_objects': 2722, 'num_object_clones': 0, 'num_object_copies': 5444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2722, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13969, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:43.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'17471", 'reported_seq': 17930, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.306625+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:41.306625+0000', 'last_peered': '2024-04-22T08:35:41.306625+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:41.306625+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11265234, 'num_objects': 2752, 'num_object_clones': 0, 'num_object_copies': 5504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17411, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'17381", 'reported_seq': 18265, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.943770+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:41.943770+0000', 'last_peered': '2024-04-22T08:35:41.943770+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:41.943770+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11490514, 'num_objects': 2807, 'num_object_clones': 0, 'num_object_copies': 5614, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17379, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -564}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17176", 'reported_seq': 19459, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.300435+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:41.300435+0000', 'last_peered': '2024-04-22T08:35:41.300435+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:41.300435+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7100", 'ondisk_log_start': "35'7100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11276288, 'num_objects': 2753, 'num_object_clones': 0, 'num_object_copies': 5506, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2687, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13630, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1065, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 66}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'17335", 'reported_seq': 18090, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.999533+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:41.999533+0000', 'last_peered': '2024-04-22T08:35:41.999533+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:41.999533+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11498089, 'num_objects': 2808, 'num_object_clones': 0, 'num_object_copies': 5616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17334, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17052", 'reported_seq': 17727, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.913132+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:41.913132+0000', 'last_peered': '2024-04-22T08:35:41.913132+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:41.913132+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7000", 'ondisk_log_start': "35'7000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11051008, 'num_objects': 2698, 'num_object_clones': 0, 'num_object_copies': 5396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13283, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'17295", 'reported_seq': 17898, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.948635+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:41.948635+0000', 'last_peered': '2024-04-22T08:35:41.948635+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:41.948635+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'7200", 'ondisk_log_start': "35'7200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11236562, 'num_objects': 2745, 'num_object_clones': 0, 'num_object_copies': 5490, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17293, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:43.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'17322", 'reported_seq': 18038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:41.984670+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:41.984670+0000', 'last_peered': '2024-04-22T08:35:41.984670+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:41.984670+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10979748, 'num_objects': 2684, 'num_object_clones': 0, 'num_object_copies': 5368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17319, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:43.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:44.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:44.871 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30832/87876 objects degraded (35.086%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":579,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":43938,"data_bytes":179920749,"bytes_used":1773424640,"bytes_avail":288136867840,"bytes_total":289910292480,"degraded_objects":30832,"degraded_total":87876,"degraded_ratio":0.3508580272201739,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":394},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:44.960 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:44.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:45.842 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:45.844 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:45.891 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-22T08:35:44.383165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1731860,"kb_used_data":536124,"kb_used_omap":12,"kb_used_meta":1195699,"kb_avail":281383660,"statfs":{"total":289910292480,"available":288136867840,"internally_reserved":0,"allocated":548990976,"data_stored":548243538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1224395886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,8,11,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19402752,"num_objects":-4737,"num_object_clones":0,"num_object_copies":-9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006581"},"pg_stats":[{"pgid":"2.f","version":"41'17427","reported_seq":18148,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:42.013261+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:42.013261+0000","last_peered":"2024-04-22T08:35:42.013261+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:42.013261+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227136,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-727}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17390","reported_seq":17511,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.319275+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:41.319275+0000","last_peered":"2024-04-22T08:35:41.319275+0000","last_clean":"2024-04-22T08:35:41.319275+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:41.319275+0000","last_undegraded":"2024-04-22T08:35:41.319275+0000","last_fullsized":"2024-04-22T08:35:41.319275+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":2842,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14446,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17142","reported_seq":17678,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.166148+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:41.166148+0000","last_peered":"2024-04-22T08:35:41.166148+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:41.166148+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11145240,"num_objects":2722,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13969,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17322","reported_seq":17357,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.350641+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:41.350641+0000","last_peered":"2024-04-22T08:35:41.350641+0000","last_clean":"2024-04-22T08:35:41.350641+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:41.350641+0000","last_undegraded":"2024-04-22T08:35:41.350641+0000","last_fullsized":"2024-04-22T08:35:41.350641+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11436032,"num_objects":2792,"num_object_clones":0,"num_object_copies":5584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17322,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17232","reported_seq":17377,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:42.012352+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:42.012352+0000","last_peered":"2024-04-22T08:35:42.012352+0000","last_clean":"2024-04-22T08:35:42.012352+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:42.012352+0000","last_undegraded":"2024-04-22T08:35:42.012352+0000","last_fullsized":"2024-04-22T08:35:42.012352+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11005952,"num_objects":2687,"num_object_clones":0,"num_object_copies":5374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14324,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17381","reported_seq":25322,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.361221+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:41.361221+0000","last_peered":"2024-04-22T08:35:41.361221+0000","last_clean":"2024-04-22T08:35:41.361221+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:41.361221+0000","last_undegraded":"2024-04-22T08:35:41.361221+0000","last_fullsized":"2024-04-22T08:35:41.361221+0000","mapping_epoch":40,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":2666,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14250,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17085","reported_seq":24827,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.373194+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:41.373194+0000","last_peered":"2024-04-22T08:35:41.373194+0000","last_clean":"2024-04-22T08:35:41.373194+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:41.373194+0000","last_undegraded":"2024-04-22T08:35:41.373194+0000","last_fullsized":"2024-04-22T08:35:41.373194+0000","mapping_epoch":36,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17092","reported_seq":17243,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.911638+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:41.911638+0000","last_peered":"2024-04-22T08:35:41.911638+0000","last_clean":"2024-04-22T08:35:41.911638+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:41.911638+0000","last_undegraded":"2024-04-22T08:35:41.911638+0000","last_fullsized":"2024-04-22T08:35:41.911638+0000","mapping_epoch":27,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":2714,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13740,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17471","reported_seq":17930,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.306625+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:41.306625+0000","last_peered":"2024-04-22T08:35:41.306625+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:41.306625+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11265234,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17411,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17381","reported_seq":18265,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.943770+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:41.943770+0000","last_peered":"2024-04-22T08:35:41.943770+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:41.943770+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11490514,"num_objects":2807,"num_object_clones":0,"num_object_copies":5614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17379,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17176","reported_seq":19459,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.300435+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:41.300435+0000","last_peered":"2024-04-22T08:35:41.300435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:41.300435+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2687,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":66}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17188","reported_seq":24869,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:41.326377+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:41.326377+0000","last_peered":"2024-04-22T08:35:41.326377+0000","last_clean":"2024-04-22T08:35:41.326377+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:41.326377+0000","last_undegraded":"2024-04-22T08:35:41.326377+0000","last_fullsized":"2024-04-22T08:35:41.326377+0000","mapping_epoch":38,"log_start":"35'7100","ondisk_log_start":"35'7100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248850,"num_objects":2748,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17186,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17335","reported_seq":18090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.999533+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:41.999533+0000","last_peered":"2024-04-22T08:35:41.999533+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:41.999533+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498089,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17334,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17052","reported_seq":17727,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.913132+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:41.913132+0000","last_peered":"2024-04-22T08:35:41.913132+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:41.913132+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7000","ondisk_log_start":"35'7000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11051008,"num_objects":2698,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13283,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17295","reported_seq":17898,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.948635+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:41.948635+0000","last_peered":"2024-04-22T08:35:41.948635+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:41.948635+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236562,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17293,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17322","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:41.984670+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:41.984670+0000","last_peered":"2024-04-22T08:35:41.984670+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:41.984670+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10979748,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17319,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":179920749,"num_objects":43938,"num_object_clones":0,"num_object_copies":87876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249766,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14775,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546803712,"data_stored":546736425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548828,"kb_used_data":210388,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93823012,"statfs":{"total":96636764160,"available":96074764288,"internally_reserved":0,"allocated":215437312,"data_stored":215200890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":2.0339999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575676,"kb_used_data":133044,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93796164,"statfs":{"total":96636764160,"available":96047271936,"internally_reserved":0,"allocated":136237056,"data_stored":135951928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":453242572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.8929999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.34000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313329,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607356,"kb_used_data":192692,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93764484,"statfs":{"total":96636764160,"available":96014831616,"internally_reserved":0,"allocated":197316608,"data_stored":197090720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":424603347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.249},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":196902912,"data_stored":196899433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135180288,"data_stored":135124031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214720512,"data_stored":214712961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:48.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:49.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:49.920 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-22T08:35:48.384016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1729776,"kb_used_data":517652,"kb_used_omap":12,"kb_used_meta":1212083,"kb_avail":281385744,"statfs":{"total":289910292480,"available":288139001856,"internally_reserved":0,"allocated":530075648,"data_stored":529328210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1241173102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,10,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23945216,"num_objects":-5846,"num_object_clones":0,"num_object_copies":-11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5846,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.f","version":"41'17594","reported_seq":18315,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.014963+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:47.014963+0000","last_peered":"2024-04-22T08:35:47.014963+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:47.014963+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17594,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17564","reported_seq":17685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.260845+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:46.260845+0000","last_peered":"2024-04-22T08:35:46.260845+0000","last_clean":"2024-04-22T08:35:46.260845+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:46.260845+0000","last_undegraded":"2024-04-22T08:35:46.260845+0000","last_fullsized":"2024-04-22T08:35:46.260845+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10928128,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14620,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17265","reported_seq":17801,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.325759+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:46.325759+0000","last_peered":"2024-04-22T08:35:46.325759+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:46.325759+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10641432,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14092,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17455","reported_seq":17491,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.323303+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:46.323303+0000","last_peered":"2024-04-22T08:35:46.323303+0000","last_clean":"2024-04-22T08:35:46.323303+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:46.323303+0000","last_undegraded":"2024-04-22T08:35:46.323303+0000","last_fullsized":"2024-04-22T08:35:46.323303+0000","mapping_epoch":27,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891264,"num_objects":2659,"num_object_clones":0,"num_object_copies":5318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17455,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17365","reported_seq":17510,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.959349+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:46.959349+0000","last_peered":"2024-04-22T08:35:46.959349+0000","last_clean":"2024-04-22T08:35:46.959349+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:46.959349+0000","last_undegraded":"2024-04-22T08:35:46.959349+0000","last_fullsized":"2024-04-22T08:35:46.959349+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10461184,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14457,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17512","reported_seq":25454,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.242013+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:46.242013+0000","last_peered":"2024-04-22T08:35:46.242013+0000","last_clean":"2024-04-22T08:35:46.242013+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:46.242013+0000","last_undegraded":"2024-04-22T08:35:46.242013+0000","last_fullsized":"2024-04-22T08:35:46.242013+0000","mapping_epoch":40,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383360,"num_objects":2535,"num_object_clones":0,"num_object_copies":5070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17242","reported_seq":24984,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.429779+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:46.429779+0000","last_peered":"2024-04-22T08:35:46.429779+0000","last_clean":"2024-04-22T08:35:46.429779+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:46.429779+0000","last_undegraded":"2024-04-22T08:35:46.429779+0000","last_fullsized":"2024-04-22T08:35:46.429779+0000","mapping_epoch":36,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739712,"num_objects":2622,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17246","reported_seq":17397,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:47.015794+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:47.015794+0000","last_peered":"2024-04-22T08:35:47.015794+0000","last_clean":"2024-04-22T08:35:47.015794+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:47.015794+0000","last_undegraded":"2024-04-22T08:35:47.015794+0000","last_fullsized":"2024-04-22T08:35:47.015794+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13894,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17609","reported_seq":18068,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.380494+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:46.380494+0000","last_peered":"2024-04-22T08:35:46.380494+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:46.380494+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10699986,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17549,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17537","reported_seq":18421,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.949776+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:46.949776+0000","last_peered":"2024-04-22T08:35:46.949776+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:46.949776+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10851538,"num_objects":2651,"num_object_clones":0,"num_object_copies":5302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17535,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17315","reported_seq":19807,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.375419+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:46.375419+0000","last_peered":"2024-04-22T08:35:46.375419+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:46.375419+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10706944,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":31}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17344","reported_seq":25025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.297980+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:46.297980+0000","last_peered":"2024-04-22T08:35:46.297980+0000","last_clean":"2024-04-22T08:35:46.297980+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:46.297980+0000","last_undegraded":"2024-04-22T08:35:46.297980+0000","last_fullsized":"2024-04-22T08:35:46.297980+0000","mapping_epoch":38,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10609874,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17342,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17462","reported_seq":18219,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.041032+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:47.041032+0000","last_peered":"2024-04-22T08:35:47.041032+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:47.041032+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10977897,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17461,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17207","reported_seq":17882,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.058155+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:47.058155+0000","last_peered":"2024-04-22T08:35:47.058155+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:47.058155+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10416128,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17433","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.823644+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:46.823644+0000","last_peered":"2024-04-22T08:35:46.823644+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:46.823644+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671314,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17431,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17445","reported_seq":18161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.033204+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:47.033204+0000","last_peered":"2024-04-22T08:35:47.033204+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:47.033204+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10475940,"num_objects":2561,"num_object_clones":0,"num_object_copies":5122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17442,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":527749120,"data_stored":527681833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117101,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548384,"kb_used_data":205848,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93823456,"statfs":{"total":96636764160,"available":96075218944,"internally_reserved":0,"allocated":210788352,"data_stored":210551930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576896,"kb_used_data":126072,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93794944,"statfs":{"total":96636764160,"available":96046022656,"internally_reserved":0,"allocated":129097728,"data_stored":128812600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":461631180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.373},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.39800000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313330,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604496,"kb_used_data":185732,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93767344,"statfs":{"total":96636764160,"available":96017760256,"internally_reserved":0,"allocated":190189568,"data_stored":189963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":428797651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.0049999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.472},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189673472,"data_stored":189669993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128110592,"data_stored":128054335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209965056,"data_stored":209957505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:49.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:50.650 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:50.665 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-22T08:35:50.384523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1729776,"kb_used_data":517652,"kb_used_omap":12,"kb_used_meta":1212083,"kb_avail":281385744,"statfs":{"total":289910292480,"available":288139001856,"internally_reserved":0,"allocated":530075648,"data_stored":529328210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1241173102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,10,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19034112,"num_objects":-4647,"num_object_clones":0,"num_object_copies":-9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4647,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002740"},"pg_stats":[{"pgid":"2.f","version":"41'17594","reported_seq":18315,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.014963+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:47.014963+0000","last_peered":"2024-04-22T08:35:47.014963+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:47.014963+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17594,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17564","reported_seq":17685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.260845+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:46.260845+0000","last_peered":"2024-04-22T08:35:46.260845+0000","last_clean":"2024-04-22T08:35:46.260845+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:46.260845+0000","last_undegraded":"2024-04-22T08:35:46.260845+0000","last_fullsized":"2024-04-22T08:35:46.260845+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10928128,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14620,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17265","reported_seq":17801,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.325759+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:46.325759+0000","last_peered":"2024-04-22T08:35:46.325759+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:46.325759+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10641432,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14092,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17455","reported_seq":17491,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.323303+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:46.323303+0000","last_peered":"2024-04-22T08:35:46.323303+0000","last_clean":"2024-04-22T08:35:46.323303+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:46.323303+0000","last_undegraded":"2024-04-22T08:35:46.323303+0000","last_fullsized":"2024-04-22T08:35:46.323303+0000","mapping_epoch":27,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891264,"num_objects":2659,"num_object_clones":0,"num_object_copies":5318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17455,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17365","reported_seq":17510,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.959349+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:46.959349+0000","last_peered":"2024-04-22T08:35:46.959349+0000","last_clean":"2024-04-22T08:35:46.959349+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:46.959349+0000","last_undegraded":"2024-04-22T08:35:46.959349+0000","last_fullsized":"2024-04-22T08:35:46.959349+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10461184,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14457,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17512","reported_seq":25454,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.242013+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:46.242013+0000","last_peered":"2024-04-22T08:35:46.242013+0000","last_clean":"2024-04-22T08:35:46.242013+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:46.242013+0000","last_undegraded":"2024-04-22T08:35:46.242013+0000","last_fullsized":"2024-04-22T08:35:46.242013+0000","mapping_epoch":40,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383360,"num_objects":2535,"num_object_clones":0,"num_object_copies":5070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17242","reported_seq":24984,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.429779+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:46.429779+0000","last_peered":"2024-04-22T08:35:46.429779+0000","last_clean":"2024-04-22T08:35:46.429779+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:46.429779+0000","last_undegraded":"2024-04-22T08:35:46.429779+0000","last_fullsized":"2024-04-22T08:35:46.429779+0000","mapping_epoch":36,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739712,"num_objects":2622,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17246","reported_seq":17397,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:47.015794+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:47.015794+0000","last_peered":"2024-04-22T08:35:47.015794+0000","last_clean":"2024-04-22T08:35:47.015794+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:47.015794+0000","last_undegraded":"2024-04-22T08:35:47.015794+0000","last_fullsized":"2024-04-22T08:35:47.015794+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13894,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17609","reported_seq":18068,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.380494+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:46.380494+0000","last_peered":"2024-04-22T08:35:46.380494+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:46.380494+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10699986,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17549,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17537","reported_seq":18421,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.949776+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:46.949776+0000","last_peered":"2024-04-22T08:35:46.949776+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:46.949776+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10851538,"num_objects":2651,"num_object_clones":0,"num_object_copies":5302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17535,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17315","reported_seq":19807,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.375419+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:46.375419+0000","last_peered":"2024-04-22T08:35:46.375419+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:46.375419+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10706944,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":31}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17344","reported_seq":25025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.297980+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:46.297980+0000","last_peered":"2024-04-22T08:35:46.297980+0000","last_clean":"2024-04-22T08:35:46.297980+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:46.297980+0000","last_undegraded":"2024-04-22T08:35:46.297980+0000","last_fullsized":"2024-04-22T08:35:46.297980+0000","mapping_epoch":38,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10609874,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17342,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17462","reported_seq":18219,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.041032+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:47.041032+0000","last_peered":"2024-04-22T08:35:47.041032+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:47.041032+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10977897,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17461,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17207","reported_seq":17882,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.058155+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:47.058155+0000","last_peered":"2024-04-22T08:35:47.058155+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:47.058155+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10416128,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17433","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.823644+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:46.823644+0000","last_peered":"2024-04-22T08:35:46.823644+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:46.823644+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671314,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17431,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17445","reported_seq":18161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.033204+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:47.033204+0000","last_peered":"2024-04-22T08:35:47.033204+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:47.033204+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10475940,"num_objects":2561,"num_object_clones":0,"num_object_copies":5122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17442,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":527749120,"data_stored":527681833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117101,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548384,"kb_used_data":205848,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93823456,"statfs":{"total":96636764160,"available":96075218944,"internally_reserved":0,"allocated":210788352,"data_stored":210551930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576896,"kb_used_data":126072,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93794944,"statfs":{"total":96636764160,"available":96046022656,"internally_reserved":0,"allocated":129097728,"data_stored":128812600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":461631180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.373},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.39800000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313330,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604496,"kb_used_data":185732,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93767344,"statfs":{"total":96636764160,"available":96017760256,"internally_reserved":0,"allocated":190189568,"data_stored":189963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":428797651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.0049999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.472},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189673472,"data_stored":189669993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128110592,"data_stored":128054335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209965056,"data_stored":209957505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:50.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:50.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'17594", 'reported_seq': 18315, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:47.014963+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:47.014963+0000', 'last_peered': '2024-04-22T08:35:47.014963+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:47.014963+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'7500", 'ondisk_log_start': "35'7500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10543104, 'num_objects': 2574, 'num_object_clones': 0, 'num_object_copies': 5148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17594, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -894}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:50.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17265", 'reported_seq': 17801, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:46.325759+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:46.325759+0000', 'last_peered': '2024-04-22T08:35:46.325759+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:46.325759+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7200", 'ondisk_log_start': "35'7200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10641432, 'num_objects': 2599, 'num_object_clones': 0, 'num_object_copies': 5198, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2599, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14092, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'17609", 'reported_seq': 18068, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:46.380494+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:46.380494+0000', 'last_peered': '2024-04-22T08:35:46.380494+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:46.380494+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'7600", 'ondisk_log_start': "35'7600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10699986, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17549, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'17537", 'reported_seq': 18421, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:46.949776+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:46.949776+0000', 'last_peered': '2024-04-22T08:35:46.949776+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:46.949776+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'7500", 'ondisk_log_start': "35'7500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10851538, 'num_objects': 2651, 'num_object_clones': 0, 'num_object_copies': 5302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17535, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -720}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17315", 'reported_seq': 19807, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:46.375419+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:46.375419+0000', 'last_peered': '2024-04-22T08:35:46.375419+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:46.375419+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10706944, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2583, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13769, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1169, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 31}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'17462", 'reported_seq': 18219, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:47.041032+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:47.041032+0000', 'last_peered': '2024-04-22T08:35:47.041032+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:47.041032+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10977897, 'num_objects': 2681, 'num_object_clones': 0, 'num_object_copies': 5362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17461, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:50.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17207", 'reported_seq': 17882, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:47.058155+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:47.058155+0000', 'last_peered': '2024-04-22T08:35:47.058155+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:47.058155+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7200", 'ondisk_log_start': "35'7200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10416128, 'num_objects': 2543, 'num_object_clones': 0, 'num_object_copies': 5086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13438, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.668 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:50.668 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'17433", 'reported_seq': 18038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:46.823644+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:46.823644+0000', 'last_peered': '2024-04-22T08:35:46.823644+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:46.823644+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10671314, 'num_objects': 2607, 'num_object_clones': 0, 'num_object_copies': 5214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17431, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.668 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:50.668 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'17445", 'reported_seq': 18161, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:47.033204+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:47.033204+0000', 'last_peered': '2024-04-22T08:35:47.033204+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:47.033204+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10475940, 'num_objects': 2561, 'num_object_clones': 0, 'num_object_copies': 5122, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17442, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:50.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:51.474 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:51.475 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30728/83268 objects degraded (36.903%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":41634,"data_bytes":170483565,"bytes_used":1771290624,"bytes_avail":288139001856,"bytes_total":289910292480,"degraded_objects":30728,"degraded_total":83268,"degraded_ratio":0.36902531584762455,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":387},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:51.548 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:51.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:52.181 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:52.223 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-22T08:35:50.384523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1729776,"kb_used_data":517652,"kb_used_omap":12,"kb_used_meta":1212083,"kb_avail":281385744,"statfs":{"total":289910292480,"available":288139001856,"internally_reserved":0,"allocated":530075648,"data_stored":529328210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1241173102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,10,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19034112,"num_objects":-4647,"num_object_clones":0,"num_object_copies":-9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4647,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002740"},"pg_stats":[{"pgid":"2.f","version":"41'17594","reported_seq":18315,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.014963+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:47.014963+0000","last_peered":"2024-04-22T08:35:47.014963+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:47.014963+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10543104,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17594,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17564","reported_seq":17685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.260845+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:46.260845+0000","last_peered":"2024-04-22T08:35:46.260845+0000","last_clean":"2024-04-22T08:35:46.260845+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:46.260845+0000","last_undegraded":"2024-04-22T08:35:46.260845+0000","last_fullsized":"2024-04-22T08:35:46.260845+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10928128,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14620,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17265","reported_seq":17801,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.325759+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:46.325759+0000","last_peered":"2024-04-22T08:35:46.325759+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:46.325759+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10641432,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14092,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17455","reported_seq":17491,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.323303+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:46.323303+0000","last_peered":"2024-04-22T08:35:46.323303+0000","last_clean":"2024-04-22T08:35:46.323303+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:46.323303+0000","last_undegraded":"2024-04-22T08:35:46.323303+0000","last_fullsized":"2024-04-22T08:35:46.323303+0000","mapping_epoch":27,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891264,"num_objects":2659,"num_object_clones":0,"num_object_copies":5318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17455,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17365","reported_seq":17510,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.959349+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:46.959349+0000","last_peered":"2024-04-22T08:35:46.959349+0000","last_clean":"2024-04-22T08:35:46.959349+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:46.959349+0000","last_undegraded":"2024-04-22T08:35:46.959349+0000","last_fullsized":"2024-04-22T08:35:46.959349+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10461184,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14457,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17512","reported_seq":25454,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.242013+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:46.242013+0000","last_peered":"2024-04-22T08:35:46.242013+0000","last_clean":"2024-04-22T08:35:46.242013+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:46.242013+0000","last_undegraded":"2024-04-22T08:35:46.242013+0000","last_fullsized":"2024-04-22T08:35:46.242013+0000","mapping_epoch":40,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383360,"num_objects":2535,"num_object_clones":0,"num_object_copies":5070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14381,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17242","reported_seq":24984,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.429779+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:46.429779+0000","last_peered":"2024-04-22T08:35:46.429779+0000","last_clean":"2024-04-22T08:35:46.429779+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:46.429779+0000","last_undegraded":"2024-04-22T08:35:46.429779+0000","last_fullsized":"2024-04-22T08:35:46.429779+0000","mapping_epoch":36,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739712,"num_objects":2622,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17246","reported_seq":17397,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:47.015794+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:47.015794+0000","last_peered":"2024-04-22T08:35:47.015794+0000","last_clean":"2024-04-22T08:35:47.015794+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:47.015794+0000","last_undegraded":"2024-04-22T08:35:47.015794+0000","last_fullsized":"2024-04-22T08:35:47.015794+0000","mapping_epoch":27,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13894,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17609","reported_seq":18068,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.380494+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:46.380494+0000","last_peered":"2024-04-22T08:35:46.380494+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:46.380494+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10699986,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17549,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17537","reported_seq":18421,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.949776+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:46.949776+0000","last_peered":"2024-04-22T08:35:46.949776+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:46.949776+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10851538,"num_objects":2651,"num_object_clones":0,"num_object_copies":5302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17535,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17315","reported_seq":19807,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.375419+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:46.375419+0000","last_peered":"2024-04-22T08:35:46.375419+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:46.375419+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10706944,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":31}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17344","reported_seq":25025,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:46.297980+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:46.297980+0000","last_peered":"2024-04-22T08:35:46.297980+0000","last_clean":"2024-04-22T08:35:46.297980+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:46.297980+0000","last_undegraded":"2024-04-22T08:35:46.297980+0000","last_fullsized":"2024-04-22T08:35:46.297980+0000","mapping_epoch":38,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10609874,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17342,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17462","reported_seq":18219,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.041032+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:47.041032+0000","last_peered":"2024-04-22T08:35:47.041032+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:47.041032+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10977897,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17461,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17207","reported_seq":17882,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.058155+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:47.058155+0000","last_peered":"2024-04-22T08:35:47.058155+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:47.058155+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7200","ondisk_log_start":"35'7200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10416128,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17433","reported_seq":18038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:46.823644+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:46.823644+0000","last_peered":"2024-04-22T08:35:46.823644+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:46.823644+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671314,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17431,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17445","reported_seq":18161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:47.033204+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:47.033204+0000","last_peered":"2024-04-22T08:35:47.033204+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:47.033204+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10475940,"num_objects":2561,"num_object_clones":0,"num_object_copies":5122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17442,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":170483565,"num_objects":41634,"num_object_clones":0,"num_object_copies":83268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252070,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14879,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":527749120,"data_stored":527681833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117101,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548384,"kb_used_data":205848,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93823456,"statfs":{"total":96636764160,"available":96075218944,"internally_reserved":0,"allocated":210788352,"data_stored":210551930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576896,"kb_used_data":126072,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93794944,"statfs":{"total":96636764160,"available":96046022656,"internally_reserved":0,"allocated":129097728,"data_stored":128812600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":461631180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.373},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.39800000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313330,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604496,"kb_used_data":185732,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93767344,"statfs":{"total":96636764160,"available":96017760256,"internally_reserved":0,"allocated":190189568,"data_stored":189963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":428797651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":2.0049999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.472},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189673472,"data_stored":189669993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128110592,"data_stored":128054335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209965056,"data_stored":209957505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:55.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:56.093 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:56.154 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-22T08:35:54.385483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161001325,"num_objects":39319,"num_object_clones":0,"num_object_copies":78638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254385,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14993,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715644,"kb_used_data":499420,"kb_used_omap":12,"kb_used_meta":1216179,"kb_avail":281399876,"statfs":{"total":289910292480,"available":288153473024,"internally_reserved":0,"allocated":511406080,"data_stored":510658642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1245367406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,11,9,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18919424,"num_objects":-4619,"num_object_clones":0,"num_object_copies":-9238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002807"},"pg_stats":[{"pgid":"2.f","version":"41'17720","reported_seq":18439,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.052473+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:52.052473+0000","last_peered":"2024-04-22T08:35:52.052473+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:52.052473+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17720,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1020}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17704","reported_seq":17824,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.325265+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:51.325265+0000","last_peered":"2024-04-22T08:35:51.325265+0000","last_clean":"2024-04-22T08:35:51.325265+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:51.325265+0000","last_undegraded":"2024-04-22T08:35:51.325265+0000","last_fullsized":"2024-04-22T08:35:51.325265+0000","mapping_epoch":27,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10354688,"num_objects":2528,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17410","reported_seq":17946,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.375917+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:51.375917+0000","last_peered":"2024-04-22T08:35:51.375917+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:51.375917+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10047512,"num_objects":2454,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14237,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17632","reported_seq":17668,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.256494+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:51.256494+0000","last_peered":"2024-04-22T08:35:51.256494+0000","last_clean":"2024-04-22T08:35:51.256494+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:51.256494+0000","last_undegraded":"2024-04-22T08:35:51.256494+0000","last_fullsized":"2024-04-22T08:35:51.256494+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10166272,"num_objects":2482,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17632,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17509","reported_seq":17654,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:52.028761+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:52.028761+0000","last_peered":"2024-04-22T08:35:52.028761+0000","last_clean":"2024-04-22T08:35:52.028761+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:52.028761+0000","last_undegraded":"2024-04-22T08:35:52.028761+0000","last_fullsized":"2024-04-22T08:35:52.028761+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9871360,"num_objects":2410,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14601,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17653","reported_seq":25595,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.509334+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:51.509334+0000","last_peered":"2024-04-22T08:35:51.509334+0000","last_clean":"2024-04-22T08:35:51.509334+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:51.509334+0000","last_undegraded":"2024-04-22T08:35:51.509334+0000","last_fullsized":"2024-04-22T08:35:51.509334+0000","mapping_epoch":40,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9805824,"num_objects":2394,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17387","reported_seq":25128,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.501275+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:51.501275+0000","last_peered":"2024-04-22T08:35:51.501275+0000","last_clean":"2024-04-22T08:35:51.501275+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:51.501275+0000","last_undegraded":"2024-04-22T08:35:51.501275+0000","last_fullsized":"2024-04-22T08:35:51.501275+0000","mapping_epoch":36,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10145792,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13755,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17379","reported_seq":17530,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.935273+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:51.935273+0000","last_peered":"2024-04-22T08:35:51.935273+0000","last_clean":"2024-04-22T08:35:51.935273+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:51.935273+0000","last_undegraded":"2024-04-22T08:35:51.935273+0000","last_fullsized":"2024-04-22T08:35:51.935273+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17758","reported_seq":18217,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.338975+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:51.338975+0000","last_peered":"2024-04-22T08:35:51.338975+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:51.338975+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089682,"num_objects":2465,"num_object_clones":0,"num_object_copies":4930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17698,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1258}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17681","reported_seq":18564,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.064604+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:52.064604+0000","last_peered":"2024-04-22T08:35:52.064604+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:52.064604+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261714,"num_objects":2507,"num_object_clones":0,"num_object_copies":5014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17679,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-864}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17446","reported_seq":20165,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.322148+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:51.322148+0000","last_peered":"2024-04-22T08:35:51.322148+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:51.322148+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10170368,"num_objects":2483,"num_object_clones":0,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13900,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17477","reported_seq":25158,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.369501+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:51.369501+0000","last_peered":"2024-04-22T08:35:51.369501+0000","last_clean":"2024-04-22T08:35:51.369501+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:51.369501+0000","last_undegraded":"2024-04-22T08:35:51.369501+0000","last_fullsized":"2024-04-22T08:35:51.369501+0000","mapping_epoch":38,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065106,"num_objects":2459,"num_object_clones":0,"num_object_copies":4918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17475,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17619","reported_seq":18373,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.051583+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:52.051583+0000","last_peered":"2024-04-22T08:35:52.051583+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:52.051583+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10334825,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17618,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-861}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17342","reported_seq":18017,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.041151+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:52.041151+0000","last_peered":"2024-04-22T08:35:52.041151+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:52.041151+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9863168,"num_objects":2408,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13573,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17592","reported_seq":18197,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.019824+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:52.019824+0000","last_peered":"2024-04-22T08:35:52.019824+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:52.019824+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10020050,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17590,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17601","reported_seq":18317,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.967538+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:51.967538+0000","last_peered":"2024-04-22T08:35:51.967538+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:51.967538+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9836964,"num_objects":2405,"num_object_clones":0,"num_object_copies":4810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17598,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161001325,"num_objects":39319,"num_object_clones":0,"num_object_copies":78638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254385,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14993,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":508837888,"data_stored":508770601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117102,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547800,"kb_used_data":201168,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93824040,"statfs":{"total":96636764160,"available":96075816960,"internally_reserved":0,"allocated":205996032,"data_stored":205759610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578316,"kb_used_data":119292,"kb_used_omap":4,"kb_used_meta":459003,"kb_avail":93793524,"statfs":{"total":96636764160,"available":96044568576,"internally_reserved":0,"allocated":122155008,"data_stored":121869880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":470019788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.7229999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.5620000000000003},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.498}]}]},{"osd":0,"up_from":21,"seq":90194313331,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589528,"kb_used_data":178960,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93782312,"statfs":{"total":96636764160,"available":96033087488,"internally_reserved":0,"allocated":183255040,"data_stored":183029152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":2.512},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.505},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.73099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182579200,"data_stored":182575721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121016320,"data_stored":120960063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205242368,"data_stored":205234817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:56.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:56.958 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:56.959 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:57.042 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-04-22T08:35:56.385911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161001325,"num_objects":39319,"num_object_clones":0,"num_object_copies":78638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254385,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14993,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715644,"kb_used_data":499420,"kb_used_omap":12,"kb_used_meta":1216179,"kb_avail":281399876,"statfs":{"total":289910292480,"available":288153473024,"internally_reserved":0,"allocated":511406080,"data_stored":510658642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1245367406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,11,9,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18919424,"num_objects":-4619,"num_object_clones":0,"num_object_copies":-9238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"2.f","version":"41'17720","reported_seq":18439,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.052473+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:52.052473+0000","last_peered":"2024-04-22T08:35:52.052473+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:52.052473+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17720,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1020}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17704","reported_seq":17824,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.325265+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:51.325265+0000","last_peered":"2024-04-22T08:35:51.325265+0000","last_clean":"2024-04-22T08:35:51.325265+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:51.325265+0000","last_undegraded":"2024-04-22T08:35:51.325265+0000","last_fullsized":"2024-04-22T08:35:51.325265+0000","mapping_epoch":27,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10354688,"num_objects":2528,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17410","reported_seq":17946,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.375917+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:51.375917+0000","last_peered":"2024-04-22T08:35:51.375917+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:51.375917+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10047512,"num_objects":2454,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14237,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17632","reported_seq":17668,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.256494+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:51.256494+0000","last_peered":"2024-04-22T08:35:51.256494+0000","last_clean":"2024-04-22T08:35:51.256494+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:51.256494+0000","last_undegraded":"2024-04-22T08:35:51.256494+0000","last_fullsized":"2024-04-22T08:35:51.256494+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10166272,"num_objects":2482,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17632,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17509","reported_seq":17654,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:52.028761+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:52.028761+0000","last_peered":"2024-04-22T08:35:52.028761+0000","last_clean":"2024-04-22T08:35:52.028761+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:52.028761+0000","last_undegraded":"2024-04-22T08:35:52.028761+0000","last_fullsized":"2024-04-22T08:35:52.028761+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9871360,"num_objects":2410,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14601,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17653","reported_seq":25595,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.509334+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:51.509334+0000","last_peered":"2024-04-22T08:35:51.509334+0000","last_clean":"2024-04-22T08:35:51.509334+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:51.509334+0000","last_undegraded":"2024-04-22T08:35:51.509334+0000","last_fullsized":"2024-04-22T08:35:51.509334+0000","mapping_epoch":40,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9805824,"num_objects":2394,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17387","reported_seq":25128,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.501275+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:51.501275+0000","last_peered":"2024-04-22T08:35:51.501275+0000","last_clean":"2024-04-22T08:35:51.501275+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:51.501275+0000","last_undegraded":"2024-04-22T08:35:51.501275+0000","last_fullsized":"2024-04-22T08:35:51.501275+0000","mapping_epoch":36,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10145792,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13755,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17379","reported_seq":17530,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.935273+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:51.935273+0000","last_peered":"2024-04-22T08:35:51.935273+0000","last_clean":"2024-04-22T08:35:51.935273+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:51.935273+0000","last_undegraded":"2024-04-22T08:35:51.935273+0000","last_fullsized":"2024-04-22T08:35:51.935273+0000","mapping_epoch":27,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582333+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:01.582333+0000","last_peered":"2024-04-22T08:35:01.582333+0000","last_clean":"2024-04-22T08:35:01.582333+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:01.582333+0000","last_undegraded":"2024-04-22T08:35:01.582333+0000","last_fullsized":"2024-04-22T08:35:01.582333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17758","reported_seq":18217,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.338975+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:51.338975+0000","last_peered":"2024-04-22T08:35:51.338975+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:51.338975+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089682,"num_objects":2465,"num_object_clones":0,"num_object_copies":4930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17698,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1258}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17681","reported_seq":18564,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.064604+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:52.064604+0000","last_peered":"2024-04-22T08:35:52.064604+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:52.064604+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261714,"num_objects":2507,"num_object_clones":0,"num_object_copies":5014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17679,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-864}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17446","reported_seq":20165,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.322148+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:51.322148+0000","last_peered":"2024-04-22T08:35:51.322148+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:51.322148+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10170368,"num_objects":2483,"num_object_clones":0,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13900,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17477","reported_seq":25158,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:51.369501+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:51.369501+0000","last_peered":"2024-04-22T08:35:51.369501+0000","last_clean":"2024-04-22T08:35:51.369501+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:51.369501+0000","last_undegraded":"2024-04-22T08:35:51.369501+0000","last_fullsized":"2024-04-22T08:35:51.369501+0000","mapping_epoch":38,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065106,"num_objects":2459,"num_object_clones":0,"num_object_copies":4918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17475,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17619","reported_seq":18373,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.051583+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:52.051583+0000","last_peered":"2024-04-22T08:35:52.051583+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:52.051583+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10334825,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17618,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-861}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17342","reported_seq":18017,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.041151+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:52.041151+0000","last_peered":"2024-04-22T08:35:52.041151+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:52.041151+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7300","ondisk_log_start":"35'7300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9863168,"num_objects":2408,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13573,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17592","reported_seq":18197,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:52.019824+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:52.019824+0000","last_peered":"2024-04-22T08:35:52.019824+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:52.019824+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10020050,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17590,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17601","reported_seq":18317,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:51.967538+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:51.967538+0000","last_peered":"2024-04-22T08:35:51.967538+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:51.967538+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9836964,"num_objects":2405,"num_object_clones":0,"num_object_copies":4810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17598,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161001325,"num_objects":39319,"num_object_clones":0,"num_object_copies":78638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254385,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14993,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":508837888,"data_stored":508770601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160710,"ondisk_log_size":160710,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117102,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547800,"kb_used_data":201168,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93824040,"statfs":{"total":96636764160,"available":96075816960,"internally_reserved":0,"allocated":205996032,"data_stored":205759610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578316,"kb_used_data":119292,"kb_used_omap":4,"kb_used_meta":459003,"kb_avail":93793524,"statfs":{"total":96636764160,"available":96044568576,"internally_reserved":0,"allocated":122155008,"data_stored":121869880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":470019788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.7229999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":4.5620000000000003},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.498}]}]},{"osd":0,"up_from":21,"seq":90194313331,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589528,"kb_used_data":178960,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93782312,"statfs":{"total":96636764160,"available":96033087488,"internally_reserved":0,"allocated":183255040,"data_stored":183029152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":420409043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":2.512},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.505},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.73099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182579200,"data_stored":182575721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121016320,"data_stored":120960063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205242368,"data_stored":205234817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'17720", 'reported_seq': 18439, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:52.052473+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:35:52.052473+0000', 'last_peered': '2024-04-22T08:35:52.052473+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:35:52.052473+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'7700", 'ondisk_log_start': "35'7700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10027008, 'num_objects': 2448, 'num_object_clones': 0, 'num_object_copies': 4896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17720, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1020}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17410", 'reported_seq': 17946, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:51.375917+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:35:51.375917+0000', 'last_peered': '2024-04-22T08:35:51.375917+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:35:51.375917+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10047512, 'num_objects': 2454, 'num_object_clones': 0, 'num_object_copies': 4908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2454, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14237, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1098}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'17758", 'reported_seq': 18217, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:51.338975+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:35:51.338975+0000', 'last_peered': '2024-04-22T08:35:51.338975+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:35:51.338975+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'7700", 'ondisk_log_start': "35'7700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10089682, 'num_objects': 2465, 'num_object_clones': 0, 'num_object_copies': 4930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17698, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:35:57.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'17681", 'reported_seq': 18564, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:52.064604+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:35:52.064604+0000', 'last_peered': '2024-04-22T08:35:52.064604+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:35:52.064604+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'7600", 'ondisk_log_start': "35'7600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10261714, 'num_objects': 2507, 'num_object_clones': 0, 'num_object_copies': 5014, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17679, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -864}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17446", 'reported_seq': 20165, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:51.322148+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:35:51.322148+0000', 'last_peered': '2024-04-22T08:35:51.322148+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:35:51.322148+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7400", 'ondisk_log_start': "35'7400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10170368, 'num_objects': 2483, 'num_object_clones': 0, 'num_object_copies': 4966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2469, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13900, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1283, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 14}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'17619", 'reported_seq': 18373, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:52.051583+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:35:52.051583+0000', 'last_peered': '2024-04-22T08:35:52.051583+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:35:52.051583+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'7600", 'ondisk_log_start': "35'7600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10334825, 'num_objects': 2524, 'num_object_clones': 0, 'num_object_copies': 5048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17618, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17342", 'reported_seq': 18017, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:52.041151+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:35:52.041151+0000', 'last_peered': '2024-04-22T08:35:52.041151+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:35:52.041151+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7300", 'ondisk_log_start': "35'7300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9863168, 'num_objects': 2408, 'num_object_clones': 0, 'num_object_copies': 4816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13573, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'17592", 'reported_seq': 18197, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:52.019824+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:35:52.019824+0000', 'last_peered': '2024-04-22T08:35:52.019824+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:35:52.019824+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'7500", 'ondisk_log_start': "35'7500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10020050, 'num_objects': 2448, 'num_object_clones': 0, 'num_object_copies': 4896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17590, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -996}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:35:57.045 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'17601", 'reported_seq': 18317, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:35:51.967538+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:35:51.967538+0000', 'last_peered': '2024-04-22T08:35:51.967538+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:35:51.967538+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'7600", 'ondisk_log_start': "35'7600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9836964, 'num_objects': 2405, 'num_object_clones': 0, 'num_object_copies': 4810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17598, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:35:57.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:35:58.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:58.034 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30614/78638 objects degraded (38.930%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":592,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":39319,"data_bytes":161001325,"bytes_used":1756819456,"bytes_avail":288153473024,"bytes_total":289910292480,"degraded_objects":30614,"degraded_total":78638,"degraded_ratio":0.38930288155853404,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":384},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:35:58.134 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:35:58.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:35:59.073 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:35:59.074 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:35:59.149 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-04-22T08:35:58.386353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151191405,"num_objects":36924,"num_object_clones":0,"num_object_copies":73848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256780,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15092,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1694496,"kb_used_data":480704,"kb_used_omap":12,"kb_used_meta":1213747,"kb_avail":281421024,"statfs":{"total":289910292480,"available":288175128576,"internally_reserved":0,"allocated":492240896,"data_stored":491493458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1242877038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,12,8,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28729344,"num_objects":-7014,"num_object_clones":0,"num_object_copies":-14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"2.f","version":"41'17863","reported_seq":18583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.044601+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:35:57.044601+0000","last_peered":"2024-04-22T08:35:57.044601+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:35:57.044601+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9441280,"num_objects":2305,"num_object_clones":0,"num_object_copies":4610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17863,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'17861","reported_seq":17982,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.374447+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:35:56.374447+0000","last_peered":"2024-04-22T08:35:56.374447+0000","last_clean":"2024-04-22T08:35:56.374447+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:35:56.374447+0000","last_undegraded":"2024-04-22T08:35:56.374447+0000","last_fullsized":"2024-04-22T08:35:56.374447+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9711616,"num_objects":2371,"num_object_clones":0,"num_object_copies":4742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14917,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17539","reported_seq":18075,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:56.339799+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:35:56.339799+0000","last_peered":"2024-04-22T08:35:56.339799+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:35:56.339799+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9519128,"num_objects":2325,"num_object_clones":0,"num_object_copies":4650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14366,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1227}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17785","reported_seq":17821,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.239963+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:35:56.239963+0000","last_peered":"2024-04-22T08:35:56.239963+0000","last_clean":"2024-04-22T08:35:56.239963+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:35:56.239963+0000","last_undegraded":"2024-04-22T08:35:56.239963+0000","last_fullsized":"2024-04-22T08:35:56.239963+0000","mapping_epoch":27,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9539584,"num_objects":2329,"num_object_clones":0,"num_object_copies":4658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17785,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17665","reported_seq":17810,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:57.099408+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:35:57.099408+0000","last_peered":"2024-04-22T08:35:57.099408+0000","last_clean":"2024-04-22T08:35:57.099408+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:35:57.099408+0000","last_undegraded":"2024-04-22T08:35:57.099408+0000","last_fullsized":"2024-04-22T08:35:57.099408+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9232384,"num_objects":2254,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17824","reported_seq":25766,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.555771+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:35:56.555771+0000","last_peered":"2024-04-22T08:35:56.555771+0000","last_clean":"2024-04-22T08:35:56.555771+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:35:56.555771+0000","last_undegraded":"2024-04-22T08:35:56.555771+0000","last_fullsized":"2024-04-22T08:35:56.555771+0000","mapping_epoch":40,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9105408,"num_objects":2223,"num_object_clones":0,"num_object_copies":4446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14693,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17535","reported_seq":25277,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.644139+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:35:56.644139+0000","last_peered":"2024-04-22T08:35:56.644139+0000","last_clean":"2024-04-22T08:35:56.644139+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:35:56.644139+0000","last_undegraded":"2024-04-22T08:35:56.644139+0000","last_fullsized":"2024-04-22T08:35:56.644139+0000","mapping_epoch":36,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9539584,"num_objects":2329,"num_object_clones":0,"num_object_copies":4658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13903,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17507","reported_seq":17658,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.879021+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:35:56.879021+0000","last_peered":"2024-04-22T08:35:56.879021+0000","last_clean":"2024-04-22T08:35:56.879021+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:35:56.879021+0000","last_undegraded":"2024-04-22T08:35:56.879021+0000","last_fullsized":"2024-04-22T08:35:56.879021+0000","mapping_epoch":27,"log_start":"35'7500","ondisk_log_start":"35'7500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9416704,"num_objects":2299,"num_object_clones":0,"num_object_copies":4598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14155,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'17893","reported_seq":18350,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:56.394786+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:35:56.394786+0000","last_peered":"2024-04-22T08:35:56.394786+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:35:56.394786+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9536722,"num_objects":2330,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17833,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1393}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17835","reported_seq":18719,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.070876+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:35:57.070876+0000","last_peered":"2024-04-22T08:35:57.070876+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:35:57.070876+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630930,"num_objects":2353,"num_object_clones":0,"num_object_copies":4706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17833,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17601","reported_seq":20519,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:56.375452+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:35:56.375452+0000","last_peered":"2024-04-22T08:35:56.375452+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:35:56.375452+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9535488,"num_objects":2328,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14055,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-42}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17636","reported_seq":25317,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:56.351300+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:35:56.351300+0000","last_peered":"2024-04-22T08:35:56.351300+0000","last_clean":"2024-04-22T08:35:56.351300+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:35:56.351300+0000","last_undegraded":"2024-04-22T08:35:56.351300+0000","last_fullsized":"2024-04-22T08:35:56.351300+0000","mapping_epoch":38,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9413842,"num_objects":2300,"num_object_clones":0,"num_object_copies":4600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17634,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17769","reported_seq":18526,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.122193+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:35:57.122193+0000","last_peered":"2024-04-22T08:35:57.122193+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:35:57.122193+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9720425,"num_objects":2374,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17768,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1011}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17496","reported_seq":18171,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.093394+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:35:57.093394+0000","last_peered":"2024-04-22T08:35:57.093394+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:35:57.093394+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7400","ondisk_log_start":"35'7400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9232384,"num_objects":2254,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13727,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17744","reported_seq":18349,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.060731+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:35:57.060731+0000","last_peered":"2024-04-22T08:35:57.060731+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:35:57.060731+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9397458,"num_objects":2296,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17742,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17752","reported_seq":18468,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:35:57.036715+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:35:57.036715+0000","last_peered":"2024-04-22T08:35:57.036715+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:35:57.036715+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9218468,"num_objects":2254,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17749,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":151191405,"num_objects":36924,"num_object_clones":0,"num_object_copies":73848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256780,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15092,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":489140224,"data_stored":489072937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117103,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530824,"kb_used_data":196480,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93841016,"statfs":{"total":96636764160,"available":96093200384,"internally_reserved":0,"allocated":201195520,"data_stored":200959098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.34200000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.38700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558988,"kb_used_data":112252,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93812852,"statfs":{"total":96636764160,"available":96064360448,"internally_reserved":0,"allocated":114946048,"data_stored":114660920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":457436876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.371},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":2.7320000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.47399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313332,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604684,"kb_used_data":171972,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93767156,"statfs":{"total":96636764160,"available":96017567744,"internally_reserved":0,"allocated":176099328,"data_stored":175873440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":443084499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.23100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175112192,"data_stored":175108713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113598464,"data_stored":113542207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200429568,"data_stored":200422017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:02.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:03.112 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:03.186 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-22T08:36:02.387291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1679692,"kb_used_data":461808,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281435828,"statfs":{"total":289910292480,"available":288190287872,"internally_reserved":0,"allocated":472891392,"data_stored":472143954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1247071342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29003776,"num_objects":-7081,"num_object_clones":0,"num_object_copies":-14162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.f","version":"41'18048","reported_seq":18767,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.215903+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:02.215903+0000","last_peered":"2024-04-22T08:36:02.215903+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:02.215903+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8683520,"num_objects":2120,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1348}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18005","reported_seq":18126,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:01.366659+0000","last_peered":"2024-04-22T08:36:01.366659+0000","last_clean":"2024-04-22T08:36:01.366659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:01.366659+0000","last_undegraded":"2024-04-22T08:36:01.366659+0000","last_fullsized":"2024-04-22T08:36:01.366659+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9121792,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17704","reported_seq":18233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.395207+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:01.395207+0000","last_peered":"2024-04-22T08:36:01.395207+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:01.395207+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8843288,"num_objects":2160,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14531,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17911","reported_seq":17947,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366720+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:01.366720+0000","last_peered":"2024-04-22T08:36:01.366720+0000","last_clean":"2024-04-22T08:36:01.366720+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:01.366720+0000","last_undegraded":"2024-04-22T08:36:01.366720+0000","last_fullsized":"2024-04-22T08:36:01.366720+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9023488,"num_objects":2203,"num_object_clones":0,"num_object_copies":4406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17911,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17809","reported_seq":17953,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.199382+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:02.199382+0000","last_peered":"2024-04-22T08:36:02.199382+0000","last_clean":"2024-04-22T08:36:02.199382+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:02.199382+0000","last_undegraded":"2024-04-22T08:36:02.199382+0000","last_fullsized":"2024-04-22T08:36:02.199382+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8642560,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17960","reported_seq":25902,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.552044+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:01.552044+0000","last_peered":"2024-04-22T08:36:01.552044+0000","last_clean":"2024-04-22T08:36:01.552044+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:01.552044+0000","last_undegraded":"2024-04-22T08:36:01.552044+0000","last_fullsized":"2024-04-22T08:36:01.552044+0000","mapping_epoch":40,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8548352,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17680","reported_seq":25422,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.607651+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:01.607651+0000","last_peered":"2024-04-22T08:36:01.607651+0000","last_clean":"2024-04-22T08:36:01.607651+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:01.607651+0000","last_undegraded":"2024-04-22T08:36:01.607651+0000","last_fullsized":"2024-04-22T08:36:01.607651+0000","mapping_epoch":36,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945664,"num_objects":2184,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17666","reported_seq":17817,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.145481+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:02.145481+0000","last_peered":"2024-04-22T08:36:02.145481+0000","last_clean":"2024-04-22T08:36:02.145481+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:02.145481+0000","last_undegraded":"2024-04-22T08:36:02.145481+0000","last_fullsized":"2024-04-22T08:36:02.145481+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8765440,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18036","reported_seq":18495,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.309466+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:01.309466+0000","last_peered":"2024-04-22T08:36:01.309466+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:01.309466+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8950994,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17976,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17975","reported_seq":18859,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.181024+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:02.181024+0000","last_peered":"2024-04-22T08:36:02.181024+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:02.181024+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17973,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17737","reported_seq":20862,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.393588+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:01.393588+0000","last_peered":"2024-04-22T08:36:01.393588+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:01.393588+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":2192,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17780","reported_seq":25461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.275782+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:01.275782+0000","last_peered":"2024-04-22T08:36:01.275782+0000","last_clean":"2024-04-22T08:36:01.275782+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:01.275782+0000","last_undegraded":"2024-04-22T08:36:01.275782+0000","last_fullsized":"2024-04-22T08:36:01.275782+0000","mapping_epoch":38,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824018,"num_objects":2156,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17778,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17916","reported_seq":18673,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.225970+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:02.225970+0000","last_peered":"2024-04-22T08:36:02.225970+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:02.225970+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9118313,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17915,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17628","reported_seq":18303,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.207083+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:02.207083+0000","last_peered":"2024-04-22T08:36:02.207083+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:02.207083+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":2122,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17902","reported_seq":18507,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.045957+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:02.045957+0000","last_peered":"2024-04-22T08:36:02.045957+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:02.045957+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8750290,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17919","reported_seq":18635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.122429+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:02.122429+0000","last_peered":"2024-04-22T08:36:02.122429+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:02.122429+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8534436,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17916,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469676032,"data_stored":469608745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117104,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530432,"kb_used_data":191992,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93841408,"statfs":{"total":96636764160,"available":96093601792,"internally_reserved":0,"allocated":196599808,"data_stored":196363386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559900,"kb_used_data":104980,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93811940,"statfs":{"total":96636764160,"available":96063426560,"internally_reserved":0,"allocated":107499520,"data_stored":107214392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.0039999999999996},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":4.1609999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313333,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589360,"kb_used_data":164836,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93782480,"statfs":{"total":96636764160,"available":96033259520,"internally_reserved":0,"allocated":168792064,"data_stored":168566176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.982},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":7.2199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167768064,"data_stored":167764585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106180608,"data_stored":106124351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195727360,"data_stored":195719809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:03.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:03.922 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:03.923 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:03.963 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-22T08:36:02.387291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1679692,"kb_used_data":461808,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281435828,"statfs":{"total":289910292480,"available":288190287872,"internally_reserved":0,"allocated":472891392,"data_stored":472143954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1247071342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29003776,"num_objects":-7081,"num_object_clones":0,"num_object_copies":-14162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.f","version":"41'18048","reported_seq":18767,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.215903+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:02.215903+0000","last_peered":"2024-04-22T08:36:02.215903+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:02.215903+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8683520,"num_objects":2120,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1348}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18005","reported_seq":18126,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:01.366659+0000","last_peered":"2024-04-22T08:36:01.366659+0000","last_clean":"2024-04-22T08:36:01.366659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:01.366659+0000","last_undegraded":"2024-04-22T08:36:01.366659+0000","last_fullsized":"2024-04-22T08:36:01.366659+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9121792,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17704","reported_seq":18233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.395207+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:01.395207+0000","last_peered":"2024-04-22T08:36:01.395207+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:01.395207+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8843288,"num_objects":2160,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14531,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17911","reported_seq":17947,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366720+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:01.366720+0000","last_peered":"2024-04-22T08:36:01.366720+0000","last_clean":"2024-04-22T08:36:01.366720+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:01.366720+0000","last_undegraded":"2024-04-22T08:36:01.366720+0000","last_fullsized":"2024-04-22T08:36:01.366720+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9023488,"num_objects":2203,"num_object_clones":0,"num_object_copies":4406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17911,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17809","reported_seq":17953,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.199382+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:02.199382+0000","last_peered":"2024-04-22T08:36:02.199382+0000","last_clean":"2024-04-22T08:36:02.199382+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:02.199382+0000","last_undegraded":"2024-04-22T08:36:02.199382+0000","last_fullsized":"2024-04-22T08:36:02.199382+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8642560,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17960","reported_seq":25902,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.552044+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:01.552044+0000","last_peered":"2024-04-22T08:36:01.552044+0000","last_clean":"2024-04-22T08:36:01.552044+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:01.552044+0000","last_undegraded":"2024-04-22T08:36:01.552044+0000","last_fullsized":"2024-04-22T08:36:01.552044+0000","mapping_epoch":40,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8548352,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17680","reported_seq":25422,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.607651+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:01.607651+0000","last_peered":"2024-04-22T08:36:01.607651+0000","last_clean":"2024-04-22T08:36:01.607651+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:01.607651+0000","last_undegraded":"2024-04-22T08:36:01.607651+0000","last_fullsized":"2024-04-22T08:36:01.607651+0000","mapping_epoch":36,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945664,"num_objects":2184,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17666","reported_seq":17817,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.145481+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:02.145481+0000","last_peered":"2024-04-22T08:36:02.145481+0000","last_clean":"2024-04-22T08:36:02.145481+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:02.145481+0000","last_undegraded":"2024-04-22T08:36:02.145481+0000","last_fullsized":"2024-04-22T08:36:02.145481+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8765440,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18036","reported_seq":18495,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.309466+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:01.309466+0000","last_peered":"2024-04-22T08:36:01.309466+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:01.309466+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8950994,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17976,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17975","reported_seq":18859,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.181024+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:02.181024+0000","last_peered":"2024-04-22T08:36:02.181024+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:02.181024+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17973,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17737","reported_seq":20862,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.393588+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:01.393588+0000","last_peered":"2024-04-22T08:36:01.393588+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:01.393588+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":2192,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17780","reported_seq":25461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.275782+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:01.275782+0000","last_peered":"2024-04-22T08:36:01.275782+0000","last_clean":"2024-04-22T08:36:01.275782+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:01.275782+0000","last_undegraded":"2024-04-22T08:36:01.275782+0000","last_fullsized":"2024-04-22T08:36:01.275782+0000","mapping_epoch":38,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824018,"num_objects":2156,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17778,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17916","reported_seq":18673,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.225970+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:02.225970+0000","last_peered":"2024-04-22T08:36:02.225970+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:02.225970+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9118313,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17915,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17628","reported_seq":18303,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.207083+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:02.207083+0000","last_peered":"2024-04-22T08:36:02.207083+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:02.207083+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":2122,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17902","reported_seq":18507,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.045957+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:02.045957+0000","last_peered":"2024-04-22T08:36:02.045957+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:02.045957+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8750290,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17919","reported_seq":18635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.122429+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:02.122429+0000","last_peered":"2024-04-22T08:36:02.122429+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:02.122429+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8534436,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17916,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469676032,"data_stored":469608745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117104,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530432,"kb_used_data":191992,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93841408,"statfs":{"total":96636764160,"available":96093601792,"internally_reserved":0,"allocated":196599808,"data_stored":196363386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559900,"kb_used_data":104980,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93811940,"statfs":{"total":96636764160,"available":96063426560,"internally_reserved":0,"allocated":107499520,"data_stored":107214392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.0039999999999996},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":4.1609999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313333,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589360,"kb_used_data":164836,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93782480,"statfs":{"total":96636764160,"available":96033259520,"internally_reserved":0,"allocated":168792064,"data_stored":168566176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.982},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":7.2199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167768064,"data_stored":167764585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106180608,"data_stored":106124351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195727360,"data_stored":195719809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18048", 'reported_seq': 18767, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.215903+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:02.215903+0000', 'last_peered': '2024-04-22T08:36:02.215903+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:02.215903+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8683520, 'num_objects': 2120, 'num_object_clones': 0, 'num_object_copies': 4240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18048, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1348}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17704", 'reported_seq': 18233, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:01.395207+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:01.395207+0000', 'last_peered': '2024-04-22T08:36:01.395207+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:01.395207+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7700", 'ondisk_log_start': "35'7700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8843288, 'num_objects': 2160, 'num_object_clones': 0, 'num_object_copies': 4320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2160, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14531, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1392}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18036", 'reported_seq': 18495, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:01.309466+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:01.309466+0000', 'last_peered': '2024-04-22T08:36:01.309466+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:01.309466+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8950994, 'num_objects': 2187, 'num_object_clones': 0, 'num_object_copies': 4374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17976, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:03.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'17975", 'reported_seq': 18859, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.181024+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:02.181024+0000', 'last_peered': '2024-04-22T08:36:02.181024+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:02.181024+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9057490, 'num_objects': 2213, 'num_object_clones': 0, 'num_object_copies': 4426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17973, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1158}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17737", 'reported_seq': 20862, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:01.393588+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:01.393588+0000', 'last_peered': '2024-04-22T08:36:01.393588+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:01.393588+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7700", 'ondisk_log_start': "35'7700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8978432, 'num_objects': 2192, 'num_object_clones': 0, 'num_object_copies': 4384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14191, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1486, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -74}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'17916", 'reported_seq': 18673, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.225970+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:02.225970+0000', 'last_peered': '2024-04-22T08:36:02.225970+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:02.225970+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9118313, 'num_objects': 2227, 'num_object_clones': 0, 'num_object_copies': 4454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17915, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1158}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17628", 'reported_seq': 18303, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.207083+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:02.207083+0000', 'last_peered': '2024-04-22T08:36:02.207083+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:02.207083+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7600", 'ondisk_log_start': "35'7600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8691712, 'num_objects': 2122, 'num_object_clones': 0, 'num_object_copies': 4244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13859, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'17902", 'reported_seq': 18507, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.045957+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:02.045957+0000', 'last_peered': '2024-04-22T08:36:02.045957+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:02.045957+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8750290, 'num_objects': 2138, 'num_object_clones': 0, 'num_object_copies': 4276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17900, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:03.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'17919", 'reported_seq': 18635, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:02.122429+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:02.122429+0000', 'last_peered': '2024-04-22T08:36:02.122429+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:02.122429+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8534436, 'num_objects': 2087, 'num_object_clones': 0, 'num_object_copies': 4174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17916, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:03.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:04.958 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:04.958 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30411/69106 objects degraded (44.006%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":599,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":34553,"data_bytes":141479789,"bytes_used":1720004608,"bytes_avail":288190287872,"bytes_total":289910292480,"degraded_objects":30411,"degraded_total":69106,"degraded_ratio":0.44006309148264983,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":397},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:36:05.016 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:05.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:05.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:05.846 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:05.907 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-04-22T08:36:04.387794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1679692,"kb_used_data":461808,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281435828,"statfs":{"total":289910292480,"available":288190287872,"internally_reserved":0,"allocated":472891392,"data_stored":472143954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1247071342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19521536,"num_objects":-4766,"num_object_clones":0,"num_object_copies":-9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002842"},"pg_stats":[{"pgid":"2.f","version":"41'18048","reported_seq":18767,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.215903+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:02.215903+0000","last_peered":"2024-04-22T08:36:02.215903+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:02.215903+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8683520,"num_objects":2120,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1348}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18005","reported_seq":18126,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366659+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:01.366659+0000","last_peered":"2024-04-22T08:36:01.366659+0000","last_clean":"2024-04-22T08:36:01.366659+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:01.366659+0000","last_undegraded":"2024-04-22T08:36:01.366659+0000","last_fullsized":"2024-04-22T08:36:01.366659+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9121792,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17704","reported_seq":18233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.395207+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:01.395207+0000","last_peered":"2024-04-22T08:36:01.395207+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:01.395207+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8843288,"num_objects":2160,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14531,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'17911","reported_seq":17947,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.366720+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:01.366720+0000","last_peered":"2024-04-22T08:36:01.366720+0000","last_clean":"2024-04-22T08:36:01.366720+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:01.366720+0000","last_undegraded":"2024-04-22T08:36:01.366720+0000","last_fullsized":"2024-04-22T08:36:01.366720+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9023488,"num_objects":2203,"num_object_clones":0,"num_object_copies":4406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17911,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17809","reported_seq":17953,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.199382+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:02.199382+0000","last_peered":"2024-04-22T08:36:02.199382+0000","last_clean":"2024-04-22T08:36:02.199382+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:02.199382+0000","last_undegraded":"2024-04-22T08:36:02.199382+0000","last_fullsized":"2024-04-22T08:36:02.199382+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8642560,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'17960","reported_seq":25902,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.552044+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:01.552044+0000","last_peered":"2024-04-22T08:36:01.552044+0000","last_clean":"2024-04-22T08:36:01.552044+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:01.552044+0000","last_undegraded":"2024-04-22T08:36:01.552044+0000","last_fullsized":"2024-04-22T08:36:01.552044+0000","mapping_epoch":40,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8548352,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14829,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17680","reported_seq":25422,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.607651+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:01.607651+0000","last_peered":"2024-04-22T08:36:01.607651+0000","last_clean":"2024-04-22T08:36:01.607651+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:01.607651+0000","last_undegraded":"2024-04-22T08:36:01.607651+0000","last_fullsized":"2024-04-22T08:36:01.607651+0000","mapping_epoch":36,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945664,"num_objects":2184,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17666","reported_seq":17817,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:02.145481+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:02.145481+0000","last_peered":"2024-04-22T08:36:02.145481+0000","last_clean":"2024-04-22T08:36:02.145481+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:02.145481+0000","last_undegraded":"2024-04-22T08:36:02.145481+0000","last_fullsized":"2024-04-22T08:36:02.145481+0000","mapping_epoch":27,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8765440,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18036","reported_seq":18495,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.309466+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:01.309466+0000","last_peered":"2024-04-22T08:36:01.309466+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:01.309466+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8950994,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17976,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'17975","reported_seq":18859,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.181024+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:02.181024+0000","last_peered":"2024-04-22T08:36:02.181024+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:02.181024+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17973,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17737","reported_seq":20862,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:01.393588+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:01.393588+0000","last_peered":"2024-04-22T08:36:01.393588+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:01.393588+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":2192,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-74}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17780","reported_seq":25461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:01.275782+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:01.275782+0000","last_peered":"2024-04-22T08:36:01.275782+0000","last_clean":"2024-04-22T08:36:01.275782+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:01.275782+0000","last_undegraded":"2024-04-22T08:36:01.275782+0000","last_fullsized":"2024-04-22T08:36:01.275782+0000","mapping_epoch":38,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824018,"num_objects":2156,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17778,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'17916","reported_seq":18673,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.225970+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:02.225970+0000","last_peered":"2024-04-22T08:36:02.225970+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:02.225970+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9118313,"num_objects":2227,"num_object_clones":0,"num_object_copies":4454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17915,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17628","reported_seq":18303,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.207083+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:02.207083+0000","last_peered":"2024-04-22T08:36:02.207083+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:02.207083+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7600","ondisk_log_start":"35'7600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":2122,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'17902","reported_seq":18507,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.045957+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:02.045957+0000","last_peered":"2024-04-22T08:36:02.045957+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:02.045957+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8750290,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'17919","reported_seq":18635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:02.122429+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:02.122429+0000","last_peered":"2024-04-22T08:36:02.122429+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:02.122429+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8534436,"num_objects":2087,"num_object_clones":0,"num_object_copies":4174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17916,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":141479789,"num_objects":34553,"num_object_clones":0,"num_object_copies":69106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259151,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15196,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469676032,"data_stored":469608745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160576,"ondisk_log_size":160576,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117104,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530432,"kb_used_data":191992,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93841408,"statfs":{"total":96636764160,"available":96093601792,"internally_reserved":0,"allocated":196599808,"data_stored":196363386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.25700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559900,"kb_used_data":104980,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93811940,"statfs":{"total":96636764160,"available":96063426560,"internally_reserved":0,"allocated":107499520,"data_stored":107214392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":465825484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":4.0039999999999996},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":4.1609999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":0.41499999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313333,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589360,"kb_used_data":164836,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93782480,"statfs":{"total":96636764160,"available":96033259520,"internally_reserved":0,"allocated":168792064,"data_stored":168566176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.982},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":7.2199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167768064,"data_stored":167764585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106180608,"data_stored":106124351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195727360,"data_stored":195719809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:08.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:09.839 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-04-22T08:36:08.389097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131981165,"num_objects":32234,"num_object_clones":0,"num_object_copies":64468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261470,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15307,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1661324,"kb_used_data":443436,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281454196,"statfs":{"total":289910292480,"available":288209096704,"internally_reserved":0,"allocated":454078464,"data_stored":453331026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1247071342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,4,6,8,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29020160,"num_objects":-7085,"num_object_clones":0,"num_object_copies":-14170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003186"},"pg_stats":[{"pgid":"2.f","version":"41'18184","reported_seq":18906,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.267559+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:07.267559+0000","last_peered":"2024-04-22T08:36:07.267559+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:07.267559+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18184,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1484}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18146","reported_seq":18267,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.404320+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:06.404320+0000","last_peered":"2024-04-22T08:36:06.404320+0000","last_clean":"2024-04-22T08:36:06.404320+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:06.404320+0000","last_undegraded":"2024-04-22T08:36:06.404320+0000","last_fullsized":"2024-04-22T08:36:06.404320+0000","mapping_epoch":27,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8544256,"num_objects":2086,"num_object_clones":0,"num_object_copies":4172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15202,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17840","reported_seq":18377,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.437130+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:06.437130+0000","last_peered":"2024-04-22T08:36:06.437130+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:06.437130+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8286232,"num_objects":2024,"num_object_clones":0,"num_object_copies":4048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14667,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18064","reported_seq":18100,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.428861+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:06.428861+0000","last_peered":"2024-04-22T08:36:06.428861+0000","last_clean":"2024-04-22T08:36:06.428861+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:06.428861+0000","last_undegraded":"2024-04-22T08:36:06.428861+0000","last_fullsized":"2024-04-22T08:36:06.428861+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8396800,"num_objects":2050,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18064,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17947","reported_seq":18092,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:07.168980+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:07.168980+0000","last_peered":"2024-04-22T08:36:07.168980+0000","last_clean":"2024-04-22T08:36:07.168980+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:07.168980+0000","last_undegraded":"2024-04-22T08:36:07.168980+0000","last_fullsized":"2024-04-22T08:36:07.168980+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8077312,"num_objects":1972,"num_object_clones":0,"num_object_copies":3944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15039,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18113","reported_seq":26055,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.777101+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:06.777101+0000","last_peered":"2024-04-22T08:36:06.777101+0000","last_clean":"2024-04-22T08:36:06.777101+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:06.777101+0000","last_undegraded":"2024-04-22T08:36:06.777101+0000","last_fullsized":"2024-04-22T08:36:06.777101+0000","mapping_epoch":40,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7921664,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14982,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17838","reported_seq":25580,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.772549+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:06.772549+0000","last_peered":"2024-04-22T08:36:06.772549+0000","last_clean":"2024-04-22T08:36:06.772549+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:06.772549+0000","last_undegraded":"2024-04-22T08:36:06.772549+0000","last_fullsized":"2024-04-22T08:36:06.772549+0000","mapping_epoch":36,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":2026,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17820","reported_seq":17971,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:07.157460+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:07.157460+0000","last_peered":"2024-04-22T08:36:07.157460+0000","last_clean":"2024-04-22T08:36:07.157460+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:07.157460+0000","last_undegraded":"2024-04-22T08:36:07.157460+0000","last_fullsized":"2024-04-22T08:36:07.157460+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134656,"num_objects":1986,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14468,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18174","reported_seq":18634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.380418+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:06.380418+0000","last_peered":"2024-04-22T08:36:06.380418+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:06.380418+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8385746,"num_objects":2049,"num_object_clones":0,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18114,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18112","reported_seq":18996,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.188657+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:07.188657+0000","last_peered":"2024-04-22T08:36:07.188657+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:07.188657+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8496338,"num_objects":2076,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18110,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1295}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17889","reported_seq":21236,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.420793+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:06.420793+0000","last_peered":"2024-04-22T08:36:06.420793+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:06.420793+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8355840,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14343,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17929","reported_seq":25610,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.425469+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:06.425469+0000","last_peered":"2024-04-22T08:36:06.425469+0000","last_clean":"2024-04-22T08:36:06.425469+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:06.425469+0000","last_undegraded":"2024-04-22T08:36:06.425469+0000","last_fullsized":"2024-04-22T08:36:06.425469+0000","mapping_epoch":38,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213714,"num_objects":2007,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17927,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18072","reported_seq":18830,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.228551+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:07.228551+0000","last_peered":"2024-04-22T08:36:07.228551+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:07.228551+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8479337,"num_objects":2071,"num_object_clones":0,"num_object_copies":4142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18071,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17764","reported_seq":18439,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.245337+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:07.245337+0000","last_peered":"2024-04-22T08:36:07.245337+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:07.245337+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134656,"num_objects":1986,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13995,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18044","reported_seq":18648,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.186547+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:07.186547+0000","last_peered":"2024-04-22T08:36:07.186547+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:07.186547+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168658,"num_objects":1996,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18042,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1448}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18059","reported_seq":18775,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.235579+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:07.235579+0000","last_peered":"2024-04-22T08:36:07.235579+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:07.235579+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7960996,"num_objects":1947,"num_object_clones":0,"num_object_copies":3894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18056,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":131981165,"num_objects":32234,"num_object_clones":0,"num_object_copies":64468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261470,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15307,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450613248,"data_stored":450545961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117105,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529828,"kb_used_data":187292,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93842012,"statfs":{"total":96636764160,"available":96094220288,"internally_reserved":0,"allocated":191787008,"data_stored":191550586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.5759999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.50700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544932,"kb_used_data":98196,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93826908,"statfs":{"total":96636764160,"available":96078753792,"internally_reserved":0,"allocated":100552704,"data_stored":100267576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":457436876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":3.0009999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":3.0339999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.1349999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313334,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586564,"kb_used_data":157948,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93785276,"statfs":{"total":96636764160,"available":96036122624,"internally_reserved":0,"allocated":161738752,"data_stored":161512864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.982},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":7.2199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160595968,"data_stored":160592489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99065856,"data_stored":99009599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190951424,"data_stored":190943873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:09.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:10.704 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:10.744 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-22T08:36:10.389608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131981165,"num_objects":32234,"num_object_clones":0,"num_object_copies":64468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261470,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15307,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1661324,"kb_used_data":443436,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281454196,"statfs":{"total":289910292480,"available":288209096704,"internally_reserved":0,"allocated":454078464,"data_stored":453331026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1247071342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,4,6,8,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19210240,"num_objects":-4690,"num_object_clones":0,"num_object_copies":-9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"2.f","version":"41'18184","reported_seq":18906,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.267559+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:07.267559+0000","last_peered":"2024-04-22T08:36:07.267559+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:07.267559+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18184,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1484}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18146","reported_seq":18267,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.404320+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:06.404320+0000","last_peered":"2024-04-22T08:36:06.404320+0000","last_clean":"2024-04-22T08:36:06.404320+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:06.404320+0000","last_undegraded":"2024-04-22T08:36:06.404320+0000","last_fullsized":"2024-04-22T08:36:06.404320+0000","mapping_epoch":27,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8544256,"num_objects":2086,"num_object_clones":0,"num_object_copies":4172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15202,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17840","reported_seq":18377,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.437130+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:06.437130+0000","last_peered":"2024-04-22T08:36:06.437130+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:06.437130+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8286232,"num_objects":2024,"num_object_clones":0,"num_object_copies":4048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14667,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18064","reported_seq":18100,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.428861+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:06.428861+0000","last_peered":"2024-04-22T08:36:06.428861+0000","last_clean":"2024-04-22T08:36:06.428861+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:06.428861+0000","last_undegraded":"2024-04-22T08:36:06.428861+0000","last_fullsized":"2024-04-22T08:36:06.428861+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8396800,"num_objects":2050,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18064,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'17947","reported_seq":18092,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:07.168980+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:07.168980+0000","last_peered":"2024-04-22T08:36:07.168980+0000","last_clean":"2024-04-22T08:36:07.168980+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:07.168980+0000","last_undegraded":"2024-04-22T08:36:07.168980+0000","last_fullsized":"2024-04-22T08:36:07.168980+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8077312,"num_objects":1972,"num_object_clones":0,"num_object_copies":3944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15039,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18113","reported_seq":26055,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.777101+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:06.777101+0000","last_peered":"2024-04-22T08:36:06.777101+0000","last_clean":"2024-04-22T08:36:06.777101+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:06.777101+0000","last_undegraded":"2024-04-22T08:36:06.777101+0000","last_fullsized":"2024-04-22T08:36:06.777101+0000","mapping_epoch":40,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7921664,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14982,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17838","reported_seq":25580,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.772549+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:06.772549+0000","last_peered":"2024-04-22T08:36:06.772549+0000","last_clean":"2024-04-22T08:36:06.772549+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:06.772549+0000","last_undegraded":"2024-04-22T08:36:06.772549+0000","last_fullsized":"2024-04-22T08:36:06.772549+0000","mapping_epoch":36,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8298496,"num_objects":2026,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17820","reported_seq":17971,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:07.157460+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:07.157460+0000","last_peered":"2024-04-22T08:36:07.157460+0000","last_clean":"2024-04-22T08:36:07.157460+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:07.157460+0000","last_undegraded":"2024-04-22T08:36:07.157460+0000","last_fullsized":"2024-04-22T08:36:07.157460+0000","mapping_epoch":27,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134656,"num_objects":1986,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14468,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18174","reported_seq":18634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.380418+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:06.380418+0000","last_peered":"2024-04-22T08:36:06.380418+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:06.380418+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8385746,"num_objects":2049,"num_object_clones":0,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18114,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18112","reported_seq":18996,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.188657+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:07.188657+0000","last_peered":"2024-04-22T08:36:07.188657+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:07.188657+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8496338,"num_objects":2076,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18110,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1295}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'17889","reported_seq":21236,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:06.420793+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:06.420793+0000","last_peered":"2024-04-22T08:36:06.420793+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:06.420793+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'7800","ondisk_log_start":"35'7800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8355840,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14343,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'17929","reported_seq":25610,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:06.425469+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:06.425469+0000","last_peered":"2024-04-22T08:36:06.425469+0000","last_clean":"2024-04-22T08:36:06.425469+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:06.425469+0000","last_undegraded":"2024-04-22T08:36:06.425469+0000","last_fullsized":"2024-04-22T08:36:06.425469+0000","mapping_epoch":38,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213714,"num_objects":2007,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17927,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18072","reported_seq":18830,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.228551+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:07.228551+0000","last_peered":"2024-04-22T08:36:07.228551+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:07.228551+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8479337,"num_objects":2071,"num_object_clones":0,"num_object_copies":4142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18071,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17764","reported_seq":18439,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.245337+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:07.245337+0000","last_peered":"2024-04-22T08:36:07.245337+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:07.245337+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7700","ondisk_log_start":"35'7700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134656,"num_objects":1986,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13995,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18044","reported_seq":18648,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.186547+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:07.186547+0000","last_peered":"2024-04-22T08:36:07.186547+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:07.186547+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168658,"num_objects":1996,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18042,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1448}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18059","reported_seq":18775,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:07.235579+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:07.235579+0000","last_peered":"2024-04-22T08:36:07.235579+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:07.235579+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7960996,"num_objects":1947,"num_object_clones":0,"num_object_copies":3894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18056,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":131981165,"num_objects":32234,"num_object_clones":0,"num_object_copies":64468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261470,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15307,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450613248,"data_stored":450545961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117105,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529828,"kb_used_data":187292,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93842012,"statfs":{"total":96636764160,"available":96094220288,"internally_reserved":0,"allocated":191787008,"data_stored":191550586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.341,"15min":1.073},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.2389999999999999,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.5759999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.4750000000000001,"15min":0.89400000000000002},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":3.2469999999999999,"5min":18.32,"15min":18.32},"last":0.50700000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544932,"kb_used_data":98196,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93826908,"statfs":{"total":96636764160,"available":96078753792,"internally_reserved":0,"allocated":100552704,"data_stored":100267576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":457436876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":0.97599999999999998,"15min":0.68600000000000005},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":9.0660000000000007,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":3.0009999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.81200000000000006,"15min":0.91000000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.0869999999999997,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":3.0339999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.5289999999999999,"15min":1.4059999999999999},"min":{"1min":0.23300000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":7.9820000000000002,"5min":15.989000000000001,"15min":15.989000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":1.7689999999999999,"5min":1.4259999999999999,"15min":1.2549999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":8.0470000000000006,"5min":20.149000000000001,"15min":20.149000000000001},"last":3.1349999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313334,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586564,"kb_used_data":157948,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93785276,"statfs":{"total":96636764160,"available":96036122624,"internally_reserved":0,"allocated":161738752,"data_stored":161512864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.514,"15min":0.78400000000000003},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.16},"max":{"1min":11.368,"5min":11.368,"15min":11.368},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.149,"15min":0.96799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":7.4400000000000004,"5min":19.658000000000001,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":1.982},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":7.2199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160595968,"data_stored":160592489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99065856,"data_stored":99009599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190951424,"data_stored":190943873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:10.745 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:10.745 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18184", 'reported_seq': 18906, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.267559+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:07.267559+0000', 'last_peered': '2024-04-22T08:36:07.267559+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:07.267559+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'8100", 'ondisk_log_start': "35'8100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8126464, 'num_objects': 1984, 'num_object_clones': 0, 'num_object_copies': 3968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18184, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1484}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17840", 'reported_seq': 18377, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:06.437130+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:06.437130+0000', 'last_peered': '2024-04-22T08:36:06.437130+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:06.437130+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7800", 'ondisk_log_start': "35'7800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8286232, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 4048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14667, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18174", 'reported_seq': 18634, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:06.380418+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:06.380418+0000', 'last_peered': '2024-04-22T08:36:06.380418+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:06.380418+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'8100", 'ondisk_log_start': "35'8100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8385746, 'num_objects': 2049, 'num_object_clones': 0, 'num_object_copies': 4098, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18114, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'18112", 'reported_seq': 18996, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.188657+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:07.188657+0000', 'last_peered': '2024-04-22T08:36:07.188657+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:07.188657+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'8100", 'ondisk_log_start': "35'8100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8496338, 'num_objects': 2076, 'num_object_clones': 0, 'num_object_copies': 4152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18110, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1295}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'17889", 'reported_seq': 21236, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:06.420793+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:06.420793+0000', 'last_peered': '2024-04-22T08:36:06.420793+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:06.420793+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'7800", 'ondisk_log_start': "35'7800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8355840, 'num_objects': 2040, 'num_object_clones': 0, 'num_object_copies': 4080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14343, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1597, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -115}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:10.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18072", 'reported_seq': 18830, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.228551+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:07.228551+0000', 'last_peered': '2024-04-22T08:36:07.228551+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:07.228551+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8479337, 'num_objects': 2071, 'num_object_clones': 0, 'num_object_copies': 4142, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18071, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17764", 'reported_seq': 18439, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.245337+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:07.245337+0000', 'last_peered': '2024-04-22T08:36:07.245337+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:07.245337+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7700", 'ondisk_log_start': "35'7700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8134656, 'num_objects': 1986, 'num_object_clones': 0, 'num_object_copies': 3972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13995, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18044", 'reported_seq': 18648, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.186547+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:07.186547+0000', 'last_peered': '2024-04-22T08:36:07.186547+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:07.186547+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8168658, 'num_objects': 1996, 'num_object_clones': 0, 'num_object_copies': 3992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18042, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1448}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:10.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18059", 'reported_seq': 18775, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:07.235579+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:07.235579+0000', 'last_peered': '2024-04-22T08:36:07.235579+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:07.235579+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7960996, 'num_objects': 1947, 'num_object_clones': 0, 'num_object_copies': 3894, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18056, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:10.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:11.709 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:11.710 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30300/64468 objects degraded (47.000%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":606,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":32234,"data_bytes":131981165,"bytes_used":1701195776,"bytes_avail":288209096704,"bytes_total":289910292480,"degraded_objects":30300,"degraded_total":64468,"degraded_ratio":0.47000062046286528,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":390},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-22T08:34:42.335545+0000","services":{}},"progress_events":{}} 2024-04-22T08:36:11.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:11.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:12.715 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:12.715 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:12.789 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-22T08:36:12.390040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1681020,"kb_used_data":424604,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281434500,"statfs":{"total":289910292480,"available":288188928000,"internally_reserved":0,"allocated":434794496,"data_stored":434047058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28897280,"num_objects":-7055,"num_object_clones":0,"num_object_copies":-14110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003208"},"pg_stats":[{"pgid":"2.f","version":"41'18331","reported_seq":19052,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.287813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:12.287813+0000","last_peered":"2024-04-22T08:36:12.287813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:12.287813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524352,"num_objects":1837,"num_object_clones":0,"num_object_copies":3674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18331,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1631}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18310","reported_seq":18431,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.430666+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:11.430666+0000","last_peered":"2024-04-22T08:36:11.430666+0000","last_clean":"2024-04-22T08:36:11.430666+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:11.430666+0000","last_undegraded":"2024-04-22T08:36:11.430666+0000","last_fullsized":"2024-04-22T08:36:11.430666+0000","mapping_epoch":27,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7872512,"num_objects":1922,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15366,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17978","reported_seq":18515,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.371766+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:11.371766+0000","last_peered":"2024-04-22T08:36:11.371766+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:11.371766+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7720984,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14805,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18203","reported_seq":18240,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.279811+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:11.279811+0000","last_peered":"2024-04-22T08:36:11.279811+0000","last_clean":"2024-04-22T08:36:11.279811+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:11.279811+0000","last_undegraded":"2024-04-22T08:36:11.279811+0000","last_fullsized":"2024-04-22T08:36:11.279811+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7827456,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18203,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18083","reported_seq":18228,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.306596+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:12.306596+0000","last_peered":"2024-04-22T08:36:12.306596+0000","last_clean":"2024-04-22T08:36:12.306596+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:12.306596+0000","last_undegraded":"2024-04-22T08:36:12.306596+0000","last_fullsized":"2024-04-22T08:36:12.306596+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15175,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18227","reported_seq":26169,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.702320+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:11.702320+0000","last_peered":"2024-04-22T08:36:11.702320+0000","last_clean":"2024-04-22T08:36:11.702320+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:11.702320+0000","last_undegraded":"2024-04-22T08:36:11.702320+0000","last_fullsized":"2024-04-22T08:36:11.702320+0000","mapping_epoch":40,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7454720,"num_objects":1820,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15096,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17977","reported_seq":25719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.798334+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:11.798334+0000","last_peered":"2024-04-22T08:36:11.798334+0000","last_clean":"2024-04-22T08:36:11.798334+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:11.798334+0000","last_undegraded":"2024-04-22T08:36:11.798334+0000","last_fullsized":"2024-04-22T08:36:11.798334+0000","mapping_epoch":36,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7729152,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14345,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17980","reported_seq":18131,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.292989+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:12.292989+0000","last_peered":"2024-04-22T08:36:12.292989+0000","last_clean":"2024-04-22T08:36:12.292989+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:12.292989+0000","last_undegraded":"2024-04-22T08:36:12.292989+0000","last_fullsized":"2024-04-22T08:36:12.292989+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479296,"num_objects":1826,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18312","reported_seq":18772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.384624+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:11.384624+0000","last_peered":"2024-04-22T08:36:11.384624+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:11.384624+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820498,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18252,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18257","reported_seq":19141,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.174680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:12.174680+0000","last_peered":"2024-04-22T08:36:12.174680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:12.174680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7902418,"num_objects":1931,"num_object_clones":0,"num_object_copies":3862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18255,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1440}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18030","reported_seq":21619,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.406316+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:11.406316+0000","last_peered":"2024-04-22T08:36:11.406316+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:11.406316+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14484,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18083","reported_seq":25764,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.327570+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:11.327570+0000","last_peered":"2024-04-22T08:36:11.327570+0000","last_clean":"2024-04-22T08:36:11.327570+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:11.327570+0000","last_undegraded":"2024-04-22T08:36:11.327570+0000","last_fullsized":"2024-04-22T08:36:11.327570+0000","mapping_epoch":38,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7582930,"num_objects":1853,"num_object_clones":0,"num_object_copies":3706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18081,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18254","reported_seq":19009,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.309771+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:12.309771+0000","last_peered":"2024-04-22T08:36:12.309771+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:12.309771+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733865,"num_objects":1889,"num_object_clones":0,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18253,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17914","reported_seq":18589,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.241866+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:12.241866+0000","last_peered":"2024-04-22T08:36:12.241866+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:12.241866+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18200","reported_seq":18805,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.244896+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:12.244896+0000","last_peered":"2024-04-22T08:36:12.244896+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:12.244896+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529682,"num_objects":1840,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18198,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18221","reported_seq":18937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.122019+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:12.122019+0000","last_peered":"2024-04-22T08:36:12.122019+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:12.122019+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7297444,"num_objects":1785,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18218,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431316992,"data_stored":431249705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117106,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529216,"kb_used_data":182584,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93842624,"statfs":{"total":96636764160,"available":96094846976,"internally_reserved":0,"allocated":186966016,"data_stored":186729594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":3.702},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":3.7320000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":3.77}]}]},{"osd":1,"up_from":33,"seq":141733920868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564048,"kb_used_data":91080,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93807792,"statfs":{"total":96636764160,"available":96059179008,"internally_reserved":0,"allocated":93265920,"data_stored":92980792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.0570000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.2000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313335,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587756,"kb_used_data":150940,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93784084,"statfs":{"total":96636764160,"available":96034902016,"internally_reserved":0,"allocated":154562560,"data_stored":154336672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.191},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.253}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.22},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":10.000999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153296896,"data_stored":153293417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91803648,"data_stored":91747391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186208897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:15.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:16.569 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:16.570 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:16.641 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-22T08:36:16.390991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1681020,"kb_used_data":424604,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281434500,"statfs":{"total":289910292480,"available":288188928000,"internally_reserved":0,"allocated":434794496,"data_stored":434047058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19185664,"num_objects":-4684,"num_object_clones":0,"num_object_copies":-9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"2.f","version":"41'18331","reported_seq":19052,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.287813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:12.287813+0000","last_peered":"2024-04-22T08:36:12.287813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:12.287813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524352,"num_objects":1837,"num_object_clones":0,"num_object_copies":3674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18331,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1631}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18310","reported_seq":18431,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.430666+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:11.430666+0000","last_peered":"2024-04-22T08:36:11.430666+0000","last_clean":"2024-04-22T08:36:11.430666+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:11.430666+0000","last_undegraded":"2024-04-22T08:36:11.430666+0000","last_fullsized":"2024-04-22T08:36:11.430666+0000","mapping_epoch":27,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7872512,"num_objects":1922,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15366,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17978","reported_seq":18515,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.371766+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:11.371766+0000","last_peered":"2024-04-22T08:36:11.371766+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:11.371766+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7720984,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14805,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18203","reported_seq":18240,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.279811+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:11.279811+0000","last_peered":"2024-04-22T08:36:11.279811+0000","last_clean":"2024-04-22T08:36:11.279811+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:11.279811+0000","last_undegraded":"2024-04-22T08:36:11.279811+0000","last_fullsized":"2024-04-22T08:36:11.279811+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7827456,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18203,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18083","reported_seq":18228,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.306596+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:12.306596+0000","last_peered":"2024-04-22T08:36:12.306596+0000","last_clean":"2024-04-22T08:36:12.306596+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:12.306596+0000","last_undegraded":"2024-04-22T08:36:12.306596+0000","last_fullsized":"2024-04-22T08:36:12.306596+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15175,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18227","reported_seq":26169,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.702320+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:11.702320+0000","last_peered":"2024-04-22T08:36:11.702320+0000","last_clean":"2024-04-22T08:36:11.702320+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:11.702320+0000","last_undegraded":"2024-04-22T08:36:11.702320+0000","last_fullsized":"2024-04-22T08:36:11.702320+0000","mapping_epoch":40,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7454720,"num_objects":1820,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15096,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17977","reported_seq":25719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.798334+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:11.798334+0000","last_peered":"2024-04-22T08:36:11.798334+0000","last_clean":"2024-04-22T08:36:11.798334+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:11.798334+0000","last_undegraded":"2024-04-22T08:36:11.798334+0000","last_fullsized":"2024-04-22T08:36:11.798334+0000","mapping_epoch":36,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7729152,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14345,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17980","reported_seq":18131,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.292989+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:12.292989+0000","last_peered":"2024-04-22T08:36:12.292989+0000","last_clean":"2024-04-22T08:36:12.292989+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:12.292989+0000","last_undegraded":"2024-04-22T08:36:12.292989+0000","last_fullsized":"2024-04-22T08:36:12.292989+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479296,"num_objects":1826,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18312","reported_seq":18772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.384624+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:11.384624+0000","last_peered":"2024-04-22T08:36:11.384624+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:11.384624+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820498,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18252,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18257","reported_seq":19141,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.174680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:12.174680+0000","last_peered":"2024-04-22T08:36:12.174680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:12.174680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7902418,"num_objects":1931,"num_object_clones":0,"num_object_copies":3862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18255,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1440}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18030","reported_seq":21619,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.406316+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:11.406316+0000","last_peered":"2024-04-22T08:36:11.406316+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:11.406316+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14484,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18083","reported_seq":25764,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.327570+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:11.327570+0000","last_peered":"2024-04-22T08:36:11.327570+0000","last_clean":"2024-04-22T08:36:11.327570+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:11.327570+0000","last_undegraded":"2024-04-22T08:36:11.327570+0000","last_fullsized":"2024-04-22T08:36:11.327570+0000","mapping_epoch":38,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7582930,"num_objects":1853,"num_object_clones":0,"num_object_copies":3706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18081,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18254","reported_seq":19009,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.309771+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:12.309771+0000","last_peered":"2024-04-22T08:36:12.309771+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:12.309771+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733865,"num_objects":1889,"num_object_clones":0,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18253,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17914","reported_seq":18589,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.241866+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:12.241866+0000","last_peered":"2024-04-22T08:36:12.241866+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:12.241866+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18200","reported_seq":18805,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.244896+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:12.244896+0000","last_peered":"2024-04-22T08:36:12.244896+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:12.244896+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529682,"num_objects":1840,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18198,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18221","reported_seq":18937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.122019+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:12.122019+0000","last_peered":"2024-04-22T08:36:12.122019+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:12.122019+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7297444,"num_objects":1785,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18218,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431316992,"data_stored":431249705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117106,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529216,"kb_used_data":182584,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93842624,"statfs":{"total":96636764160,"available":96094846976,"internally_reserved":0,"allocated":186966016,"data_stored":186729594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":3.702},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":3.7320000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":3.77}]}]},{"osd":1,"up_from":33,"seq":141733920868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564048,"kb_used_data":91080,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93807792,"statfs":{"total":96636764160,"available":96059179008,"internally_reserved":0,"allocated":93265920,"data_stored":92980792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.0570000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.2000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313335,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587756,"kb_used_data":150940,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93784084,"statfs":{"total":96636764160,"available":96034902016,"internally_reserved":0,"allocated":154562560,"data_stored":154336672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.191},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.253}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.22},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":10.000999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153296896,"data_stored":153293417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91803648,"data_stored":91747391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186208897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:16.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:17.540 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-22T08:36:16.390991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1681020,"kb_used_data":424604,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281434500,"statfs":{"total":289910292480,"available":288188928000,"internally_reserved":0,"allocated":434794496,"data_stored":434047058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19185664,"num_objects":-4684,"num_object_clones":0,"num_object_copies":-9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"2.f","version":"41'18331","reported_seq":19052,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.287813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:12.287813+0000","last_peered":"2024-04-22T08:36:12.287813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:12.287813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524352,"num_objects":1837,"num_object_clones":0,"num_object_copies":3674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18331,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1631}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18310","reported_seq":18431,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.430666+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:11.430666+0000","last_peered":"2024-04-22T08:36:11.430666+0000","last_clean":"2024-04-22T08:36:11.430666+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:11.430666+0000","last_undegraded":"2024-04-22T08:36:11.430666+0000","last_fullsized":"2024-04-22T08:36:11.430666+0000","mapping_epoch":27,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7872512,"num_objects":1922,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15366,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'17978","reported_seq":18515,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.371766+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:11.371766+0000","last_peered":"2024-04-22T08:36:11.371766+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:11.371766+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7720984,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14805,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1666}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18203","reported_seq":18240,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.279811+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:11.279811+0000","last_peered":"2024-04-22T08:36:11.279811+0000","last_clean":"2024-04-22T08:36:11.279811+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:11.279811+0000","last_undegraded":"2024-04-22T08:36:11.279811+0000","last_fullsized":"2024-04-22T08:36:11.279811+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7827456,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18203,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18083","reported_seq":18228,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.306596+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:12.306596+0000","last_peered":"2024-04-22T08:36:12.306596+0000","last_clean":"2024-04-22T08:36:12.306596+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:12.306596+0000","last_undegraded":"2024-04-22T08:36:12.306596+0000","last_fullsized":"2024-04-22T08:36:12.306596+0000","mapping_epoch":27,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15175,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18227","reported_seq":26169,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.702320+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:11.702320+0000","last_peered":"2024-04-22T08:36:11.702320+0000","last_clean":"2024-04-22T08:36:11.702320+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:11.702320+0000","last_undegraded":"2024-04-22T08:36:11.702320+0000","last_fullsized":"2024-04-22T08:36:11.702320+0000","mapping_epoch":40,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7454720,"num_objects":1820,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15096,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'17977","reported_seq":25719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.798334+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:11.798334+0000","last_peered":"2024-04-22T08:36:11.798334+0000","last_clean":"2024-04-22T08:36:11.798334+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:11.798334+0000","last_undegraded":"2024-04-22T08:36:11.798334+0000","last_fullsized":"2024-04-22T08:36:11.798334+0000","mapping_epoch":36,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7729152,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14345,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'17980","reported_seq":18131,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:12.292989+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:12.292989+0000","last_peered":"2024-04-22T08:36:12.292989+0000","last_clean":"2024-04-22T08:36:12.292989+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:12.292989+0000","last_undegraded":"2024-04-22T08:36:12.292989+0000","last_fullsized":"2024-04-22T08:36:12.292989+0000","mapping_epoch":27,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479296,"num_objects":1826,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18312","reported_seq":18772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.384624+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:11.384624+0000","last_peered":"2024-04-22T08:36:11.384624+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:11.384624+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820498,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18252,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18257","reported_seq":19141,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.174680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:12.174680+0000","last_peered":"2024-04-22T08:36:12.174680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:12.174680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7902418,"num_objects":1931,"num_object_clones":0,"num_object_copies":3862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18255,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1440}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18030","reported_seq":21619,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:11.406316+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:11.406316+0000","last_peered":"2024-04-22T08:36:11.406316+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:11.406316+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14484,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.581952+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:01.581952+0000","last_peered":"2024-04-22T08:35:01.581952+0000","last_clean":"2024-04-22T08:35:01.581952+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:01.581952+0000","last_undegraded":"2024-04-22T08:35:01.581952+0000","last_fullsized":"2024-04-22T08:35:01.581952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18083","reported_seq":25764,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:11.327570+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:11.327570+0000","last_peered":"2024-04-22T08:36:11.327570+0000","last_clean":"2024-04-22T08:36:11.327570+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:11.327570+0000","last_undegraded":"2024-04-22T08:36:11.327570+0000","last_fullsized":"2024-04-22T08:36:11.327570+0000","mapping_epoch":38,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7582930,"num_objects":1853,"num_object_clones":0,"num_object_copies":3706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18081,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18254","reported_seq":19009,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.309771+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:12.309771+0000","last_peered":"2024-04-22T08:36:12.309771+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:12.309771+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733865,"num_objects":1889,"num_object_clones":0,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18253,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582114+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:01.582114+0000","last_peered":"2024-04-22T08:35:01.582114+0000","last_clean":"2024-04-22T08:35:01.582114+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:01.582114+0000","last_undegraded":"2024-04-22T08:35:01.582114+0000","last_fullsized":"2024-04-22T08:35:01.582114+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'17914","reported_seq":18589,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.241866+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:12.241866+0000","last_peered":"2024-04-22T08:36:12.241866+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:12.241866+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'7900","ondisk_log_start":"35'7900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18200","reported_seq":18805,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.244896+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:12.244896+0000","last_peered":"2024-04-22T08:36:12.244896+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:12.244896+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529682,"num_objects":1840,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18198,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.582199+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:01.582199+0000","last_peered":"2024-04-22T08:35:01.582199+0000","last_clean":"2024-04-22T08:35:01.582199+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:01.582199+0000","last_undegraded":"2024-04-22T08:35:01.582199+0000","last_fullsized":"2024-04-22T08:35:01.582199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18221","reported_seq":18937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:12.122019+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:12.122019+0000","last_peered":"2024-04-22T08:36:12.122019+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:12.122019+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7297444,"num_objects":1785,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18218,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":122294125,"num_objects":29869,"num_object_clones":0,"num_object_copies":59738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263835,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15429,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":431316992,"data_stored":431249705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117106,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529216,"kb_used_data":182584,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93842624,"statfs":{"total":96636764160,"available":96094846976,"internally_reserved":0,"allocated":186966016,"data_stored":186729594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":3.702},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":3.7320000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":3.77}]}]},{"osd":1,"up_from":33,"seq":141733920868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564048,"kb_used_data":91080,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93807792,"statfs":{"total":96636764160,"available":96059179008,"internally_reserved":0,"allocated":93265920,"data_stored":92980792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.0570000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.2000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313335,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587756,"kb_used_data":150940,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93784084,"statfs":{"total":96636764160,"available":96034902016,"internally_reserved":0,"allocated":154562560,"data_stored":154336672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.191},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.253}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.22},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":10.000999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153296896,"data_stored":153293417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91803648,"data_stored":91747391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186208897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:17.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:17.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18331", 'reported_seq': 19052, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.287813+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:12.287813+0000', 'last_peered': '2024-04-22T08:36:12.287813+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:12.287813+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "35'8300", 'ondisk_log_start': "35'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7524352, 'num_objects': 1837, 'num_object_clones': 0, 'num_object_copies': 3674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18331, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1631}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'17978", 'reported_seq': 18515, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:11.371766+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:11.371766+0000', 'last_peered': '2024-04-22T08:36:11.371766+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:11.371766+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7720984, 'num_objects': 1886, 'num_object_clones': 0, 'num_object_copies': 3772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1886, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14805, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18312", 'reported_seq': 18772, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:11.384624+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:11.384624+0000', 'last_peered': '2024-04-22T08:36:11.384624+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:11.384624+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "35'8300", 'ondisk_log_start': "35'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7820498, 'num_objects': 1911, 'num_object_clones': 0, 'num_object_copies': 3822, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18252, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'18257", 'reported_seq': 19141, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.174680+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:12.174680+0000', 'last_peered': '2024-04-22T08:36:12.174680+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:12.174680+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "35'8200", 'ondisk_log_start': "35'8200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7902418, 'num_objects': 1931, 'num_object_clones': 0, 'num_object_copies': 3862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18255, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1440}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'18030", 'reported_seq': 21619, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:11.406316+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:11.406316+0000', 'last_peered': '2024-04-22T08:36:11.406316+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:11.406316+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7778304, 'num_objects': 1899, 'num_object_clones': 0, 'num_object_copies': 3798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14484, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1719, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:17.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18254", 'reported_seq': 19009, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.309771+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:12.309771+0000', 'last_peered': '2024-04-22T08:36:12.309771+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:12.309771+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "35'8200", 'ondisk_log_start': "35'8200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7733865, 'num_objects': 1889, 'num_object_clones': 0, 'num_object_copies': 3778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18253, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'17914", 'reported_seq': 18589, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.241866+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:12.241866+0000', 'last_peered': '2024-04-22T08:36:12.241866+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:12.241866+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'7900", 'ondisk_log_start': "35'7900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7520256, 'num_objects': 1836, 'num_object_clones': 0, 'num_object_copies': 3672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14145, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18200", 'reported_seq': 18805, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.244896+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:12.244896+0000', 'last_peered': '2024-04-22T08:36:12.244896+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:12.244896+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'8100", 'ondisk_log_start': "35'8100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7529682, 'num_objects': 1840, 'num_object_clones': 0, 'num_object_copies': 3680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18198, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:17.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18221", 'reported_seq': 18937, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:12.122019+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:12.122019+0000', 'last_peered': '2024-04-22T08:36:12.122019+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:12.122019+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'8200", 'ondisk_log_start': "35'8200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7297444, 'num_objects': 1785, 'num_object_clones': 0, 'num_object_copies': 3570, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18218, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:17.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:18.523 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:18.524 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30059/54994 objects degraded (54.659%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":613,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":27497,"data_bytes":112578413,"bytes_used":1664225280,"bytes_avail":288246067200,"bytes_total":289910292480,"degraded_objects":30059,"degraded_total":54994,"degraded_ratio":0.54658690038913338,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":352,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":587},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:18.574 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:18.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:19.376 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:19.377 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:19.475 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-04-22T08:36:18.391463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112578413,"num_objects":27497,"num_object_clones":0,"num_object_copies":54994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266207,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15548,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1625220,"kb_used_data":405676,"kb_used_omap":12,"kb_used_meta":1219507,"kb_avail":281490300,"statfs":{"total":289910292480,"available":288246067200,"internally_reserved":0,"allocated":415412224,"data_stored":414664786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1248775278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,2,3,8,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28901376,"num_objects":-7056,"num_object_clones":0,"num_object_copies":-14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003239"},"pg_stats":[{"pgid":"2.f","version":"41'18459","reported_seq":19181,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.312258+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:17.312258+0000","last_peered":"2024-04-22T08:36:17.312258+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:17.312258+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7000064,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18459,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18456","reported_seq":18577,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:16.402064+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:16.402064+0000","last_peered":"2024-04-22T08:36:16.402064+0000","last_clean":"2024-04-22T08:36:16.402064+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:16.402064+0000","last_undegraded":"2024-04-22T08:36:16.402064+0000","last_fullsized":"2024-04-22T08:36:16.402064+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":1776,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15512,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18128","reported_seq":18665,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:16.297829+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:16.297829+0000","last_peered":"2024-04-22T08:36:16.297829+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:16.297829+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7106584,"num_objects":1736,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14955,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18359","reported_seq":18396,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:16.444225+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:16.444225+0000","last_peered":"2024-04-22T08:36:16.444225+0000","last_clean":"2024-04-22T08:36:16.444225+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:16.444225+0000","last_undegraded":"2024-04-22T08:36:16.444225+0000","last_fullsized":"2024-04-22T08:36:16.444225+0000","mapping_epoch":27,"log_start":"36'8300","ondisk_log_start":"36'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7188480,"num_objects":1755,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18359,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18246","reported_seq":18391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.307548+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:17.307548+0000","last_peered":"2024-04-22T08:36:17.307548+0000","last_clean":"2024-04-22T08:36:17.307548+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:17.307548+0000","last_undegraded":"2024-04-22T08:36:17.307548+0000","last_fullsized":"2024-04-22T08:36:17.307548+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15338,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18361","reported_seq":26303,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:16.804797+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:16.804797+0000","last_peered":"2024-04-22T08:36:16.804797+0000","last_clean":"2024-04-22T08:36:16.804797+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:16.804797+0000","last_undegraded":"2024-04-22T08:36:16.804797+0000","last_fullsized":"2024-04-22T08:36:16.804797+0000","mapping_epoch":40,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6905856,"num_objects":1686,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15230,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18125","reported_seq":25867,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:16.852411+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:16.852411+0000","last_peered":"2024-04-22T08:36:16.852411+0000","last_clean":"2024-04-22T08:36:16.852411+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:16.852411+0000","last_undegraded":"2024-04-22T08:36:16.852411+0000","last_fullsized":"2024-04-22T08:36:16.852411+0000","mapping_epoch":36,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7122944,"num_objects":1739,"num_object_clones":0,"num_object_copies":3478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14493,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18122","reported_seq":18272,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.375634+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:17.375634+0000","last_peered":"2024-04-22T08:36:17.375634+0000","last_clean":"2024-04-22T08:36:17.375634+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:17.375634+0000","last_undegraded":"2024-04-22T08:36:17.375634+0000","last_fullsized":"2024-04-22T08:36:17.375634+0000","mapping_epoch":27,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6897664,"num_objects":1684,"num_object_clones":0,"num_object_copies":3368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14770,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18465","reported_seq":18924,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:16.437766+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:16.437766+0000","last_peered":"2024-04-22T08:36:16.437766+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:16.437766+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"36'8400","ondisk_log_start":"36'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193810,"num_objects":1758,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18405,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18414","reported_seq":19298,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.259704+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:17.259704+0000","last_peered":"2024-04-22T08:36:17.259704+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:17.259704+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7259346,"num_objects":1774,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18412,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18166","reported_seq":21993,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:16.401044+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:16.401044+0000","last_peered":"2024-04-22T08:36:16.401044+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:16.401044+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7221248,"num_objects":1763,"num_object_clones":0,"num_object_copies":3526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14620,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-151}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18230","reported_seq":25911,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:16.373034+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:16.373034+0000","last_peered":"2024-04-22T08:36:16.373034+0000","last_clean":"2024-04-22T08:36:16.373034+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:16.373034+0000","last_undegraded":"2024-04-22T08:36:16.373034+0000","last_fullsized":"2024-04-22T08:36:16.373034+0000","mapping_epoch":38,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6980818,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18228,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18401","reported_seq":19159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.379997+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:17.379997+0000","last_peered":"2024-04-22T08:36:17.379997+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:17.379997+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7131753,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18400,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18076","reported_seq":18751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.393578+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:17.393578+0000","last_peered":"2024-04-22T08:36:17.393578+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:17.393578+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6856704,"num_objects":1674,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14307,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18354","reported_seq":18959,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.367949+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:17.367949+0000","last_peered":"2024-04-22T08:36:17.367949+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:17.367949+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898898,"num_objects":1686,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18352,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18370","reported_seq":19086,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.376059+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:17.376059+0000","last_peered":"2024-04-22T08:36:17.376059+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:17.376059+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6687140,"num_objects":1636,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18367,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":112578413,"num_objects":27497,"num_object_clones":0,"num_object_copies":54994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266207,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15548,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":411910144,"data_stored":411842857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117107,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512132,"kb_used_data":177788,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93859708,"statfs":{"total":96636764160,"available":96112340992,"internally_reserved":0,"allocated":182054912,"data_stored":181818490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.208,"15min":0.97599999999999998},"min":{"1min":0.222,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.2809999999999997,"5min":13.635,"15min":13.635},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.78600000000000003,"15min":0.72599999999999998},"min":{"1min":0.249,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":3.4630000000000001,"5min":13.721,"15min":13.721},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":3.4900000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544764,"kb_used_data":84084,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93827076,"statfs":{"total":96636764160,"available":96078925824,"internally_reserved":0,"allocated":86102016,"data_stored":85816888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":471723724},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":6.0570000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.2000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313336,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568324,"kb_used_data":143804,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93803516,"statfs":{"total":96636764160,"available":96054800384,"internally_reserved":0,"allocated":147255296,"data_stored":147029408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146026496,"data_stored":146023017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84574208,"data_stored":84517951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181309440,"data_stored":181301889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:22.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:23.172 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:23.172 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:23.216 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-04-22T08:36:22.392412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107536237,"num_objects":26266,"num_object_clones":0,"num_object_copies":52532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267438,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1625744,"kb_used_data":393904,"kb_used_omap":12,"kb_used_meta":1231795,"kb_avail":281489776,"statfs":{"total":289910292480,"available":288245530624,"internally_reserved":0,"allocated":403357696,"data_stored":402610258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1261358190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24444928,"num_objects":-5968,"num_object_clones":0,"num_object_copies":-11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"2.f","version":"41'18459","reported_seq":19181,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.312258+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:17.312258+0000","last_peered":"2024-04-22T08:36:17.312258+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:17.312258+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7000064,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18459,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18598","reported_seq":18719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.475197+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:21.475197+0000","last_peered":"2024-04-22T08:36:21.475197+0000","last_clean":"2024-04-22T08:36:21.475197+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:21.475197+0000","last_undegraded":"2024-04-22T08:36:21.475197+0000","last_fullsized":"2024-04-22T08:36:21.475197+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":1634,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15654,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18278","reported_seq":18815,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439677+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:21.439677+0000","last_peered":"2024-04-22T08:36:21.439677+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:21.439677+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"36'8200","ondisk_log_start":"36'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6492184,"num_objects":1586,"num_object_clones":0,"num_object_copies":3172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15105,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18499","reported_seq":18535,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.464025+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:21.464025+0000","last_peered":"2024-04-22T08:36:21.464025+0000","last_clean":"2024-04-22T08:36:21.464025+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:21.464025+0000","last_undegraded":"2024-04-22T08:36:21.464025+0000","last_fullsized":"2024-04-22T08:36:21.464025+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6615040,"num_objects":1615,"num_object_clones":0,"num_object_copies":3230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18499,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18246","reported_seq":18391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.307548+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:17.307548+0000","last_peered":"2024-04-22T08:36:17.307548+0000","last_clean":"2024-04-22T08:36:17.307548+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:17.307548+0000","last_undegraded":"2024-04-22T08:36:17.307548+0000","last_fullsized":"2024-04-22T08:36:17.307548+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15338,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18516","reported_seq":26458,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.913440+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:21.913440+0000","last_peered":"2024-04-22T08:36:21.913440+0000","last_clean":"2024-04-22T08:36:21.913440+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:21.913440+0000","last_undegraded":"2024-04-22T08:36:21.913440+0000","last_fullsized":"2024-04-22T08:36:21.913440+0000","mapping_epoch":40,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15385,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18293","reported_seq":26035,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.835486+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:21.835486+0000","last_peered":"2024-04-22T08:36:21.835486+0000","last_clean":"2024-04-22T08:36:21.835486+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:21.835486+0000","last_undegraded":"2024-04-22T08:36:21.835486+0000","last_fullsized":"2024-04-22T08:36:21.835486+0000","mapping_epoch":36,"log_start":"37'8200","ondisk_log_start":"37'8200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6434816,"num_objects":1571,"num_object_clones":0,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14661,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18122","reported_seq":18272,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.375634+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:17.375634+0000","last_peered":"2024-04-22T08:36:17.375634+0000","last_clean":"2024-04-22T08:36:17.375634+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:17.375634+0000","last_undegraded":"2024-04-22T08:36:17.375634+0000","last_fullsized":"2024-04-22T08:36:17.375634+0000","mapping_epoch":27,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6897664,"num_objects":1684,"num_object_clones":0,"num_object_copies":3368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14770,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18630","reported_seq":19090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.405987+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:21.405987+0000","last_peered":"2024-04-22T08:36:21.405987+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:21.405987+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6517970,"num_objects":1593,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18570,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18414","reported_seq":19298,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.259704+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:17.259704+0000","last_peered":"2024-04-22T08:36:17.259704+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:17.259704+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7259346,"num_objects":1774,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18412,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18332","reported_seq":22364,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439812+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:21.439812+0000","last_peered":"2024-04-22T08:36:21.439812+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:21.439812+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6541312,"num_objects":1597,"num_object_clones":0,"num_object_copies":3194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14786,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18375","reported_seq":26056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.301824+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:21.301824+0000","last_peered":"2024-04-22T08:36:21.301824+0000","last_clean":"2024-04-22T08:36:21.301824+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:21.301824+0000","last_undegraded":"2024-04-22T08:36:21.301824+0000","last_fullsized":"2024-04-22T08:36:21.301824+0000","mapping_epoch":38,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386898,"num_objects":1561,"num_object_clones":0,"num_object_copies":3122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18373,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18401","reported_seq":19159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.379997+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:17.379997+0000","last_peered":"2024-04-22T08:36:17.379997+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:17.379997+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7131753,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18400,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18076","reported_seq":18751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.393578+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:17.393578+0000","last_peered":"2024-04-22T08:36:17.393578+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:17.393578+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6856704,"num_objects":1674,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14307,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18354","reported_seq":18959,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.367949+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:17.367949+0000","last_peered":"2024-04-22T08:36:17.367949+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:17.367949+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898898,"num_objects":1686,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18352,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18370","reported_seq":19086,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.376059+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:17.376059+0000","last_peered":"2024-04-22T08:36:17.376059+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:17.376059+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6687140,"num_objects":1636,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18367,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":107536237,"num_objects":26266,"num_object_clones":0,"num_object_copies":52532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267438,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399695872,"data_stored":399628585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117108,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511508,"kb_used_data":173068,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93860332,"statfs":{"total":96636764160,"available":96112979968,"internally_reserved":0,"allocated":177221632,"data_stored":176985210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":4.9119999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":4.9420000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.35499999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545912,"kb_used_data":77032,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93825928,"statfs":{"total":96636764160,"available":96077750272,"internally_reserved":0,"allocated":78880768,"data_stored":78595640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313336,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568324,"kb_used_data":143804,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93803516,"statfs":{"total":96636764160,"available":96054800384,"internally_reserved":0,"allocated":147255296,"data_stored":147029408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146026496,"data_stored":146023017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77156352,"data_stored":77100095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176513024,"data_stored":176505473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:23.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:23.998 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:23.999 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:24.059 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-04-22T08:36:22.392412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107536237,"num_objects":26266,"num_object_clones":0,"num_object_copies":52532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267438,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1625744,"kb_used_data":393904,"kb_used_omap":12,"kb_used_meta":1231795,"kb_avail":281489776,"statfs":{"total":289910292480,"available":288245530624,"internally_reserved":0,"allocated":403357696,"data_stored":402610258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1261358190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24444928,"num_objects":-5968,"num_object_clones":0,"num_object_copies":-11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"2.f","version":"41'18459","reported_seq":19181,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.312258+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:17.312258+0000","last_peered":"2024-04-22T08:36:17.312258+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:17.312258+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7000064,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18459,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18598","reported_seq":18719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.475197+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:21.475197+0000","last_peered":"2024-04-22T08:36:21.475197+0000","last_clean":"2024-04-22T08:36:21.475197+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:21.475197+0000","last_undegraded":"2024-04-22T08:36:21.475197+0000","last_fullsized":"2024-04-22T08:36:21.475197+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":1634,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15654,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18278","reported_seq":18815,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439677+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:21.439677+0000","last_peered":"2024-04-22T08:36:21.439677+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:21.439677+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"36'8200","ondisk_log_start":"36'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6492184,"num_objects":1586,"num_object_clones":0,"num_object_copies":3172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15105,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18499","reported_seq":18535,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.464025+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:21.464025+0000","last_peered":"2024-04-22T08:36:21.464025+0000","last_clean":"2024-04-22T08:36:21.464025+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:21.464025+0000","last_undegraded":"2024-04-22T08:36:21.464025+0000","last_fullsized":"2024-04-22T08:36:21.464025+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6615040,"num_objects":1615,"num_object_clones":0,"num_object_copies":3230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18499,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18246","reported_seq":18391,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.307548+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:17.307548+0000","last_peered":"2024-04-22T08:36:17.307548+0000","last_clean":"2024-04-22T08:36:17.307548+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:17.307548+0000","last_undegraded":"2024-04-22T08:36:17.307548+0000","last_fullsized":"2024-04-22T08:36:17.307548+0000","mapping_epoch":27,"log_start":"35'8200","ondisk_log_start":"35'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15338,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18516","reported_seq":26458,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.913440+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:21.913440+0000","last_peered":"2024-04-22T08:36:21.913440+0000","last_clean":"2024-04-22T08:36:21.913440+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:21.913440+0000","last_undegraded":"2024-04-22T08:36:21.913440+0000","last_fullsized":"2024-04-22T08:36:21.913440+0000","mapping_epoch":40,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15385,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18293","reported_seq":26035,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.835486+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:21.835486+0000","last_peered":"2024-04-22T08:36:21.835486+0000","last_clean":"2024-04-22T08:36:21.835486+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:21.835486+0000","last_undegraded":"2024-04-22T08:36:21.835486+0000","last_fullsized":"2024-04-22T08:36:21.835486+0000","mapping_epoch":36,"log_start":"37'8200","ondisk_log_start":"37'8200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6434816,"num_objects":1571,"num_object_clones":0,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14661,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18122","reported_seq":18272,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:17.375634+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:17.375634+0000","last_peered":"2024-04-22T08:36:17.375634+0000","last_clean":"2024-04-22T08:36:17.375634+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:17.375634+0000","last_undegraded":"2024-04-22T08:36:17.375634+0000","last_fullsized":"2024-04-22T08:36:17.375634+0000","mapping_epoch":27,"log_start":"35'8100","ondisk_log_start":"35'8100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6897664,"num_objects":1684,"num_object_clones":0,"num_object_copies":3368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14770,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18630","reported_seq":19090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.405987+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:21.405987+0000","last_peered":"2024-04-22T08:36:21.405987+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:21.405987+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6517970,"num_objects":1593,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18570,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18414","reported_seq":19298,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.259704+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:17.259704+0000","last_peered":"2024-04-22T08:36:17.259704+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:17.259704+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7259346,"num_objects":1774,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18412,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18332","reported_seq":22364,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439812+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:21.439812+0000","last_peered":"2024-04-22T08:36:21.439812+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:21.439812+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6541312,"num_objects":1597,"num_object_clones":0,"num_object_copies":3194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14786,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18375","reported_seq":26056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.301824+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:21.301824+0000","last_peered":"2024-04-22T08:36:21.301824+0000","last_clean":"2024-04-22T08:36:21.301824+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:21.301824+0000","last_undegraded":"2024-04-22T08:36:21.301824+0000","last_fullsized":"2024-04-22T08:36:21.301824+0000","mapping_epoch":38,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386898,"num_objects":1561,"num_object_clones":0,"num_object_copies":3122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18373,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18401","reported_seq":19159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.379997+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:17.379997+0000","last_peered":"2024-04-22T08:36:17.379997+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:17.379997+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7131753,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18400,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18076","reported_seq":18751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.393578+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:17.393578+0000","last_peered":"2024-04-22T08:36:17.393578+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:17.393578+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"35'8000","ondisk_log_start":"35'8000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6856704,"num_objects":1674,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14307,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18354","reported_seq":18959,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.367949+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:17.367949+0000","last_peered":"2024-04-22T08:36:17.367949+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:17.367949+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898898,"num_objects":1686,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18352,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18370","reported_seq":19086,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:17.376059+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:17.376059+0000","last_peered":"2024-04-22T08:36:17.376059+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:17.376059+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"35'8300","ondisk_log_start":"35'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6687140,"num_objects":1636,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18367,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":107536237,"num_objects":26266,"num_object_clones":0,"num_object_copies":52532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267438,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399695872,"data_stored":399628585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117108,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511508,"kb_used_data":173068,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93860332,"statfs":{"total":96636764160,"available":96112979968,"internally_reserved":0,"allocated":177221632,"data_stored":176985210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":4.9119999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":4.9420000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.35499999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545912,"kb_used_data":77032,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93825928,"statfs":{"total":96636764160,"available":96077750272,"internally_reserved":0,"allocated":78880768,"data_stored":78595640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313336,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568324,"kb_used_data":143804,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93803516,"statfs":{"total":96636764160,"available":96054800384,"internally_reserved":0,"allocated":147255296,"data_stored":147029408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146026496,"data_stored":146023017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77156352,"data_stored":77100095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176513024,"data_stored":176505473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:24.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:24.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18459", 'reported_seq': 19181, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.312258+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:17.312258+0000', 'last_peered': '2024-04-22T08:36:17.312258+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:17.312258+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7000064, 'num_objects': 1709, 'num_object_clones': 0, 'num_object_copies': 3418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18459, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1759}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:24.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'18278", 'reported_seq': 18815, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:21.439677+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:21.439677+0000', 'last_peered': '2024-04-22T08:36:21.439677+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:21.439677+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "36'8200", 'ondisk_log_start': "36'8200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6492184, 'num_objects': 1586, 'num_object_clones': 0, 'num_object_copies': 3172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1586, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15105, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18630", 'reported_seq': 19090, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:21.405987+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:21.405987+0000', 'last_peered': '2024-04-22T08:36:21.405987+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:21.405987+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'8600", 'ondisk_log_start': "37'8600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6517970, 'num_objects': 1593, 'num_object_clones': 0, 'num_object_copies': 3186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18570, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'18414", 'reported_seq': 19298, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.259704+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:17.259704+0000', 'last_peered': '2024-04-22T08:36:17.259704+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:17.259704+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7259346, 'num_objects': 1774, 'num_object_clones': 0, 'num_object_copies': 3548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18412, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1597}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'18332", 'reported_seq': 22364, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:21.439812+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:21.439812+0000', 'last_peered': '2024-04-22T08:36:21.439812+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:21.439812+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'8300", 'ondisk_log_start': "37'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6541312, 'num_objects': 1597, 'num_object_clones': 0, 'num_object_copies': 3194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1813, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14786, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1939, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -216}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:24.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18401", 'reported_seq': 19159, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.379997+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:17.379997+0000', 'last_peered': '2024-04-22T08:36:17.379997+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:17.379997+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7131753, 'num_objects': 1742, 'num_object_clones': 0, 'num_object_copies': 3484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18400, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'18076", 'reported_seq': 18751, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.393578+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:17.393578+0000', 'last_peered': '2024-04-22T08:36:17.393578+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:17.393578+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "35'8000", 'ondisk_log_start': "35'8000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6856704, 'num_objects': 1674, 'num_object_clones': 0, 'num_object_copies': 3348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14307, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1916}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18354", 'reported_seq': 18959, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.367949+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:17.367949+0000', 'last_peered': '2024-04-22T08:36:17.367949+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:17.367949+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "35'8300", 'ondisk_log_start': "35'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6898898, 'num_objects': 1686, 'num_object_clones': 0, 'num_object_copies': 3372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18352, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:24.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18370", 'reported_seq': 19086, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:17.376059+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:17.376059+0000', 'last_peered': '2024-04-22T08:36:17.376059+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:17.376059+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "35'8300", 'ondisk_log_start': "35'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6687140, 'num_objects': 1636, 'num_object_clones': 0, 'num_object_copies': 3272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18367, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:24.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:25.104 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29958/50250 objects degraded (59.618%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":620,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":25125,"data_bytes":102862701,"bytes_used":1666060288,"bytes_avail":288244232192,"bytes_total":289910292480,"degraded_objects":29958,"degraded_total":50250,"degraded_ratio":0.59617910447761191,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":395},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:25.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:25.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:25.437 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:25.438 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:25.453 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-22T08:36:24.392881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102862701,"num_objects":25125,"num_object_clones":0,"num_object_copies":50250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25125,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268579,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1627012,"kb_used_data":386980,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281488508,"statfs":{"total":289910292480,"available":288244232192,"internally_reserved":0,"allocated":396267520,"data_stored":395520082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1269746798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,11,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19431424,"num_objects":-4744,"num_object_clones":0,"num_object_copies":-9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4744,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"2.f","version":"41'18610","reported_seq":19332,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.357261+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:22.357261+0000","last_peered":"2024-04-22T08:36:22.357261+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:22.357261+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6381568,"num_objects":1558,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18610,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18598","reported_seq":18719,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.475197+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:21.475197+0000","last_peered":"2024-04-22T08:36:21.475197+0000","last_clean":"2024-04-22T08:36:21.475197+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:21.475197+0000","last_undegraded":"2024-04-22T08:36:21.475197+0000","last_fullsized":"2024-04-22T08:36:21.475197+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":1634,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15654,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18278","reported_seq":18815,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439677+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:21.439677+0000","last_peered":"2024-04-22T08:36:21.439677+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:21.439677+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"36'8200","ondisk_log_start":"36'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6492184,"num_objects":1586,"num_object_clones":0,"num_object_copies":3172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15105,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18499","reported_seq":18535,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.464025+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:21.464025+0000","last_peered":"2024-04-22T08:36:21.464025+0000","last_clean":"2024-04-22T08:36:21.464025+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:21.464025+0000","last_undegraded":"2024-04-22T08:36:21.464025+0000","last_fullsized":"2024-04-22T08:36:21.464025+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6615040,"num_objects":1615,"num_object_clones":0,"num_object_copies":3230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18499,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18367","reported_seq":18512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:22.384461+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:22.384461+0000","last_peered":"2024-04-22T08:36:22.384461+0000","last_clean":"2024-04-22T08:36:22.384461+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:22.384461+0000","last_undegraded":"2024-04-22T08:36:22.384461+0000","last_fullsized":"2024-04-22T08:36:22.384461+0000","mapping_epoch":27,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":1552,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15459,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18516","reported_seq":26458,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.913440+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:21.913440+0000","last_peered":"2024-04-22T08:36:21.913440+0000","last_clean":"2024-04-22T08:36:21.913440+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:21.913440+0000","last_undegraded":"2024-04-22T08:36:21.913440+0000","last_fullsized":"2024-04-22T08:36:21.913440+0000","mapping_epoch":40,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15385,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18293","reported_seq":26035,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.835486+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:21.835486+0000","last_peered":"2024-04-22T08:36:21.835486+0000","last_clean":"2024-04-22T08:36:21.835486+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:21.835486+0000","last_undegraded":"2024-04-22T08:36:21.835486+0000","last_fullsized":"2024-04-22T08:36:21.835486+0000","mapping_epoch":36,"log_start":"37'8200","ondisk_log_start":"37'8200","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6434816,"num_objects":1571,"num_object_clones":0,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14661,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18275","reported_seq":18426,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:22.371900+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:22.371900+0000","last_peered":"2024-04-22T08:36:22.371900+0000","last_clean":"2024-04-22T08:36:22.371900+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:22.371900+0000","last_undegraded":"2024-04-22T08:36:22.371900+0000","last_fullsized":"2024-04-22T08:36:22.371900+0000","mapping_epoch":27,"log_start":"36'8200","ondisk_log_start":"36'8200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14923,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18630","reported_seq":19090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.405987+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:21.405987+0000","last_peered":"2024-04-22T08:36:21.405987+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:21.405987+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6517970,"num_objects":1593,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18570,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18567","reported_seq":19451,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.218073+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:22.218073+0000","last_peered":"2024-04-22T08:36:22.218073+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:22.218073+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6632658,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18565,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1750}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18332","reported_seq":22364,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:21.439812+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:21.439812+0000","last_peered":"2024-04-22T08:36:21.439812+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:21.439812+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6541312,"num_objects":1597,"num_object_clones":0,"num_object_copies":3194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14786,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18375","reported_seq":26056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:21.301824+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:21.301824+0000","last_peered":"2024-04-22T08:36:21.301824+0000","last_clean":"2024-04-22T08:36:21.301824+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:21.301824+0000","last_undegraded":"2024-04-22T08:36:21.301824+0000","last_fullsized":"2024-04-22T08:36:21.301824+0000","mapping_epoch":38,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386898,"num_objects":1561,"num_object_clones":0,"num_object_copies":3122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18373,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18554","reported_seq":19311,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.396536+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:22.396536+0000","last_peered":"2024-04-22T08:36:22.396536+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:22.396536+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6505065,"num_objects":1589,"num_object_clones":0,"num_object_copies":3178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18553,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1796}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18211","reported_seq":18886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.355968+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:22.355968+0000","last_peered":"2024-04-22T08:36:22.355968+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:22.355968+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8200","ondisk_log_start":"37'8200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6303744,"num_objects":1539,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14442,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18494","reported_seq":19098,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.331620+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:22.331620+0000","last_peered":"2024-04-22T08:36:22.331620+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:22.331620+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6325458,"num_objects":1546,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18492,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18505","reported_seq":19222,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:22.302367+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:22.302367+0000","last_peered":"2024-04-22T08:36:22.302367+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:22.302367+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134180,"num_objects":1501,"num_object_clones":0,"num_object_copies":3002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18502,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":102862701,"num_objects":25125,"num_object_clones":0,"num_object_copies":50250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25125,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268579,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15649,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392421376,"data_stored":392354089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117108,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511508,"kb_used_data":173068,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93860332,"statfs":{"total":96636764160,"available":96112979968,"internally_reserved":0,"allocated":177221632,"data_stored":176985210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":4.9119999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":4.9420000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.35499999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545912,"kb_used_data":77032,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93825928,"statfs":{"total":96636764160,"available":96077750272,"internally_reserved":0,"allocated":78880768,"data_stored":78595640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313337,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569592,"kb_used_data":136880,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93802248,"statfs":{"total":96636764160,"available":96053501952,"internally_reserved":0,"allocated":140165120,"data_stored":139939232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":443084499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.1020000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.7729999999999997,"5min":8,"15min":8},"last":0.44},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.0609999999999999,"15min":0.89300000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.5,"5min":12.131,"15min":12.131},"last":0.41599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":138752000,"data_stored":138748521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77156352,"data_stored":77100095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176513024,"data_stored":176505473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:28.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:29.297 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:29.301 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:29.345 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-22T08:36:28.393786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1626016,"kb_used_data":369608,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281489504,"statfs":{"total":289910292480,"available":288245252096,"internally_reserved":0,"allocated":378478592,"data_stored":377731154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,3,8,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28160000,"num_objects":-6875,"num_object_clones":0,"num_object_copies":-13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.f","version":"41'18741","reported_seq":19463,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.198651+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:27.198651+0000","last_peered":"2024-04-22T08:36:27.198651+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:27.198651+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5844992,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18741,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18751","reported_seq":18872,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.424717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:26.424717+0000","last_peered":"2024-04-22T08:36:26.424717+0000","last_clean":"2024-04-22T08:36:26.424717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:26.424717+0000","last_undegraded":"2024-04-22T08:36:26.424717+0000","last_fullsized":"2024-04-22T08:36:26.424717+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6066176,"num_objects":1481,"num_object_clones":0,"num_object_copies":2962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15807,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18421","reported_seq":18958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.435100+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:26.435100+0000","last_peered":"2024-04-22T08:36:26.435100+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:26.435100+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906456,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15248,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18648","reported_seq":18685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.451676+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:26.451676+0000","last_peered":"2024-04-22T08:36:26.451676+0000","last_clean":"2024-04-22T08:36:26.451676+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:26.451676+0000","last_undegraded":"2024-04-22T08:36:26.451676+0000","last_fullsized":"2024-04-22T08:36:26.451676+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18476","reported_seq":18621,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.350318+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:27.350318+0000","last_peered":"2024-04-22T08:36:27.350318+0000","last_clean":"2024-04-22T08:36:27.350318+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:27.350318+0000","last_undegraded":"2024-04-22T08:36:27.350318+0000","last_fullsized":"2024-04-22T08:36:27.350318+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5910528,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15568,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18629","reported_seq":26571,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.904222+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:26.904222+0000","last_peered":"2024-04-22T08:36:26.904222+0000","last_clean":"2024-04-22T08:36:26.904222+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:26.904222+0000","last_undegraded":"2024-04-22T08:36:26.904222+0000","last_fullsized":"2024-04-22T08:36:26.904222+0000","mapping_epoch":40,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5808128,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15498,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18440","reported_seq":26182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.987729+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:26.987729+0000","last_peered":"2024-04-22T08:36:26.987729+0000","last_clean":"2024-04-22T08:36:26.987729+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:26.987729+0000","last_undegraded":"2024-04-22T08:36:26.987729+0000","last_fullsized":"2024-04-22T08:36:26.987729+0000","mapping_epoch":36,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":1424,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18389","reported_seq":18540,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.390011+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:27.390011+0000","last_peered":"2024-04-22T08:36:27.390011+0000","last_clean":"2024-04-22T08:36:27.390011+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:27.390011+0000","last_undegraded":"2024-04-22T08:36:27.390011+0000","last_fullsized":"2024-04-22T08:36:27.390011+0000","mapping_epoch":27,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18756","reported_seq":19216,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.432490+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:26.432490+0000","last_peered":"2024-04-22T08:36:26.432490+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:26.432490+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6001874,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18696,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18711","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.405428+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:27.405428+0000","last_peered":"2024-04-22T08:36:27.405428+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:27.405428+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6042834,"num_objects":1477,"num_object_clones":0,"num_object_copies":2954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18709,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18479","reported_seq":22727,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.476655+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:26.476655+0000","last_peered":"2024-04-22T08:36:26.476655+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:26.476655+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5939200,"num_objects":1450,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14933,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18497","reported_seq":26178,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.402248+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:26.402248+0000","last_peered":"2024-04-22T08:36:26.402248+0000","last_clean":"2024-04-22T08:36:26.402248+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:26.402248+0000","last_undegraded":"2024-04-22T08:36:26.402248+0000","last_fullsized":"2024-04-22T08:36:26.402248+0000","mapping_epoch":38,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887186,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18495,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18698","reported_seq":19455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.367112+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:27.367112+0000","last_peered":"2024-04-22T08:36:27.367112+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:27.367112+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915241,"num_objects":1445,"num_object_clones":0,"num_object_copies":2890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18697,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18340","reported_seq":19016,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.400928+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:27.400928+0000","last_peered":"2024-04-22T08:36:27.400928+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:27.400928+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5775360,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18621","reported_seq":19226,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.377080+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:27.377080+0000","last_peered":"2024-04-22T08:36:27.377080+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:27.377080+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18619,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18638","reported_seq":19355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.319976+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:27.319976+0000","last_peered":"2024-04-22T08:36:27.319976+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:27.319976+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5589412,"num_objects":1368,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18635,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374960128,"data_stored":374892841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117109,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511208,"kb_used_data":168672,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93860632,"statfs":{"total":96636764160,"available":96113287168,"internally_reserved":0,"allocated":172720128,"data_stored":172483706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.191},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.36099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547468,"kb_used_data":70404,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93824372,"statfs":{"total":96636764160,"available":96076156928,"internally_reserved":0,"allocated":72093696,"data_stored":71808568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":488500940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313338,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567340,"kb_used_data":130532,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93804500,"statfs":{"total":96636764160,"available":96055808000,"internally_reserved":0,"allocated":133664768,"data_stored":133438880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.133},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132317184,"data_stored":132313705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70549504,"data_stored":70493247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172093440,"data_stored":172085889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:29.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:29.790 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:29.836 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-22T08:36:28.393786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1626016,"kb_used_data":369608,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281489504,"statfs":{"total":289910292480,"available":288245252096,"internally_reserved":0,"allocated":378478592,"data_stored":377731154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,3,8,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28160000,"num_objects":-6875,"num_object_clones":0,"num_object_copies":-13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.f","version":"41'18741","reported_seq":19463,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.198651+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:27.198651+0000","last_peered":"2024-04-22T08:36:27.198651+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:27.198651+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5844992,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18741,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18751","reported_seq":18872,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.424717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:26.424717+0000","last_peered":"2024-04-22T08:36:26.424717+0000","last_clean":"2024-04-22T08:36:26.424717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:26.424717+0000","last_undegraded":"2024-04-22T08:36:26.424717+0000","last_fullsized":"2024-04-22T08:36:26.424717+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6066176,"num_objects":1481,"num_object_clones":0,"num_object_copies":2962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15807,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18421","reported_seq":18958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.435100+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:26.435100+0000","last_peered":"2024-04-22T08:36:26.435100+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:26.435100+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906456,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15248,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18648","reported_seq":18685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.451676+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:26.451676+0000","last_peered":"2024-04-22T08:36:26.451676+0000","last_clean":"2024-04-22T08:36:26.451676+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:26.451676+0000","last_undegraded":"2024-04-22T08:36:26.451676+0000","last_fullsized":"2024-04-22T08:36:26.451676+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18476","reported_seq":18621,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.350318+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:27.350318+0000","last_peered":"2024-04-22T08:36:27.350318+0000","last_clean":"2024-04-22T08:36:27.350318+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:27.350318+0000","last_undegraded":"2024-04-22T08:36:27.350318+0000","last_fullsized":"2024-04-22T08:36:27.350318+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5910528,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15568,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18629","reported_seq":26571,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.904222+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:26.904222+0000","last_peered":"2024-04-22T08:36:26.904222+0000","last_clean":"2024-04-22T08:36:26.904222+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:26.904222+0000","last_undegraded":"2024-04-22T08:36:26.904222+0000","last_fullsized":"2024-04-22T08:36:26.904222+0000","mapping_epoch":40,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5808128,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15498,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18440","reported_seq":26182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.987729+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:26.987729+0000","last_peered":"2024-04-22T08:36:26.987729+0000","last_clean":"2024-04-22T08:36:26.987729+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:26.987729+0000","last_undegraded":"2024-04-22T08:36:26.987729+0000","last_fullsized":"2024-04-22T08:36:26.987729+0000","mapping_epoch":36,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":1424,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18389","reported_seq":18540,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.390011+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:27.390011+0000","last_peered":"2024-04-22T08:36:27.390011+0000","last_clean":"2024-04-22T08:36:27.390011+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:27.390011+0000","last_undegraded":"2024-04-22T08:36:27.390011+0000","last_fullsized":"2024-04-22T08:36:27.390011+0000","mapping_epoch":27,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18756","reported_seq":19216,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.432490+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:26.432490+0000","last_peered":"2024-04-22T08:36:26.432490+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:26.432490+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6001874,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18696,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18711","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.405428+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:27.405428+0000","last_peered":"2024-04-22T08:36:27.405428+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:27.405428+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6042834,"num_objects":1477,"num_object_clones":0,"num_object_copies":2954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18709,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18479","reported_seq":22727,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.476655+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:26.476655+0000","last_peered":"2024-04-22T08:36:26.476655+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:26.476655+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5939200,"num_objects":1450,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14933,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18497","reported_seq":26178,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.402248+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:26.402248+0000","last_peered":"2024-04-22T08:36:26.402248+0000","last_clean":"2024-04-22T08:36:26.402248+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:26.402248+0000","last_undegraded":"2024-04-22T08:36:26.402248+0000","last_fullsized":"2024-04-22T08:36:26.402248+0000","mapping_epoch":38,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887186,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18495,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18698","reported_seq":19455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.367112+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:27.367112+0000","last_peered":"2024-04-22T08:36:27.367112+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:27.367112+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915241,"num_objects":1445,"num_object_clones":0,"num_object_copies":2890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18697,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18340","reported_seq":19016,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.400928+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:27.400928+0000","last_peered":"2024-04-22T08:36:27.400928+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:27.400928+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5775360,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18621","reported_seq":19226,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.377080+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:27.377080+0000","last_peered":"2024-04-22T08:36:27.377080+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:27.377080+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18619,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18638","reported_seq":19355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.319976+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:27.319976+0000","last_peered":"2024-04-22T08:36:27.319976+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:27.319976+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5589412,"num_objects":1368,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18635,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374960128,"data_stored":374892841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117109,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511208,"kb_used_data":168672,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93860632,"statfs":{"total":96636764160,"available":96113287168,"internally_reserved":0,"allocated":172720128,"data_stored":172483706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.191},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.36099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547468,"kb_used_data":70404,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93824372,"statfs":{"total":96636764160,"available":96076156928,"internally_reserved":0,"allocated":72093696,"data_stored":71808568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":488500940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313338,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567340,"kb_used_data":130532,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93804500,"statfs":{"total":96636764160,"available":96055808000,"internally_reserved":0,"allocated":133664768,"data_stored":133438880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.133},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132317184,"data_stored":132313705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70549504,"data_stored":70493247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172093440,"data_stored":172085889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:29.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:29.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18741", 'reported_seq': 19463, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.198651+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:27.198651+0000', 'last_peered': '2024-04-22T08:36:27.198651+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:27.198651+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'8700", 'ondisk_log_start': "37'8700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5844992, 'num_objects': 1427, 'num_object_clones': 0, 'num_object_copies': 2854, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18741, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2041}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'18421", 'reported_seq': 18958, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:26.435100+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:26.435100+0000', 'last_peered': '2024-04-22T08:36:26.435100+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:26.435100+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5906456, 'num_objects': 1443, 'num_object_clones': 0, 'num_object_copies': 2886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1443, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15248, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18756", 'reported_seq': 19216, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:26.432490+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:26.432490+0000', 'last_peered': '2024-04-22T08:36:26.432490+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:26.432490+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'8700", 'ondisk_log_start': "37'8700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6001874, 'num_objects': 1467, 'num_object_clones': 0, 'num_object_copies': 2934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18696, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'18711", 'reported_seq': 19594, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.405428+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:27.405428+0000', 'last_peered': '2024-04-22T08:36:27.405428+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:27.405428+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'8700", 'ondisk_log_start': "37'8700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6042834, 'num_objects': 1477, 'num_object_clones': 0, 'num_object_copies': 2954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18709, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1894}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'18479", 'reported_seq': 22727, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:26.476655+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:26.476655+0000', 'last_peered': '2024-04-22T08:36:26.476655+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:26.476655+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5939200, 'num_objects': 1450, 'num_object_clones': 0, 'num_object_copies': 2900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1705, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14933, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2047, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -255}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:29.838 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18698", 'reported_seq': 19455, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.367112+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:27.367112+0000', 'last_peered': '2024-04-22T08:36:27.367112+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:27.367112+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'8600", 'ondisk_log_start': "37'8600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5915241, 'num_objects': 1445, 'num_object_clones': 0, 'num_object_copies': 2890, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18697, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'18340", 'reported_seq': 19016, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.400928+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:27.400928+0000', 'last_peered': '2024-04-22T08:36:27.400928+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:27.400928+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'8300", 'ondisk_log_start': "37'8300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5775360, 'num_objects': 1410, 'num_object_clones': 0, 'num_object_copies': 2820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14571, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18621", 'reported_seq': 19226, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.377080+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:27.377080+0000', 'last_peered': '2024-04-22T08:36:27.377080+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:27.377080+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'8600", 'ondisk_log_start': "37'8600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5805266, 'num_objects': 1419, 'num_object_clones': 0, 'num_object_copies': 2838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18619, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:29.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18638", 'reported_seq': 19355, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:27.319976+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:27.319976+0000', 'last_peered': '2024-04-22T08:36:27.319976+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:27.319976+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'8600", 'ondisk_log_start': "37'8600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5589412, 'num_objects': 1368, 'num_object_clones': 0, 'num_object_copies': 2736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18635, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:29.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:30.928 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:30.928 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29850/45988 objects degraded (64.908%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22994,"data_bytes":94134125,"bytes_used":1665040384,"bytes_avail":288245252096,"bytes_total":289910292480,"degraded_objects":29850,"degraded_total":45988,"degraded_ratio":0.64908236931373398,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:31.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:31.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:31.753 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:31.755 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:31.811 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-22T08:36:30.394318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1626016,"kb_used_data":369608,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281489504,"statfs":{"total":289910292480,"available":288245252096,"internally_reserved":0,"allocated":378478592,"data_stored":377731154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1286524014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,3,8,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18444288,"num_objects":-4503,"num_object_clones":0,"num_object_copies":-9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002855"},"pg_stats":[{"pgid":"2.f","version":"41'18741","reported_seq":19463,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.198651+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:27.198651+0000","last_peered":"2024-04-22T08:36:27.198651+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:27.198651+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5844992,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18741,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18751","reported_seq":18872,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.424717+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:26.424717+0000","last_peered":"2024-04-22T08:36:26.424717+0000","last_clean":"2024-04-22T08:36:26.424717+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:26.424717+0000","last_undegraded":"2024-04-22T08:36:26.424717+0000","last_fullsized":"2024-04-22T08:36:26.424717+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6066176,"num_objects":1481,"num_object_clones":0,"num_object_copies":2962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15807,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18421","reported_seq":18958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.435100+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:26.435100+0000","last_peered":"2024-04-22T08:36:26.435100+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:26.435100+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906456,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15248,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18648","reported_seq":18685,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.451676+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:26.451676+0000","last_peered":"2024-04-22T08:36:26.451676+0000","last_clean":"2024-04-22T08:36:26.451676+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:26.451676+0000","last_undegraded":"2024-04-22T08:36:26.451676+0000","last_fullsized":"2024-04-22T08:36:26.451676+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18476","reported_seq":18621,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.350318+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:27.350318+0000","last_peered":"2024-04-22T08:36:27.350318+0000","last_clean":"2024-04-22T08:36:27.350318+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:27.350318+0000","last_undegraded":"2024-04-22T08:36:27.350318+0000","last_fullsized":"2024-04-22T08:36:27.350318+0000","mapping_epoch":27,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5910528,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15568,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18629","reported_seq":26571,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.904222+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:26.904222+0000","last_peered":"2024-04-22T08:36:26.904222+0000","last_clean":"2024-04-22T08:36:26.904222+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:26.904222+0000","last_undegraded":"2024-04-22T08:36:26.904222+0000","last_fullsized":"2024-04-22T08:36:26.904222+0000","mapping_epoch":40,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5808128,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15498,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18440","reported_seq":26182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.987729+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:26.987729+0000","last_peered":"2024-04-22T08:36:26.987729+0000","last_clean":"2024-04-22T08:36:26.987729+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:26.987729+0000","last_undegraded":"2024-04-22T08:36:26.987729+0000","last_fullsized":"2024-04-22T08:36:26.987729+0000","mapping_epoch":36,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":1424,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18389","reported_seq":18540,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:27.390011+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:27.390011+0000","last_peered":"2024-04-22T08:36:27.390011+0000","last_clean":"2024-04-22T08:36:27.390011+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:27.390011+0000","last_undegraded":"2024-04-22T08:36:27.390011+0000","last_fullsized":"2024-04-22T08:36:27.390011+0000","mapping_epoch":27,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18756","reported_seq":19216,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.432490+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:26.432490+0000","last_peered":"2024-04-22T08:36:26.432490+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:26.432490+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6001874,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18696,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18711","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.405428+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:27.405428+0000","last_peered":"2024-04-22T08:36:27.405428+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:27.405428+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6042834,"num_objects":1477,"num_object_clones":0,"num_object_copies":2954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18709,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18479","reported_seq":22727,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:26.476655+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:26.476655+0000","last_peered":"2024-04-22T08:36:26.476655+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:26.476655+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5939200,"num_objects":1450,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14933,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18497","reported_seq":26178,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:26.402248+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:26.402248+0000","last_peered":"2024-04-22T08:36:26.402248+0000","last_clean":"2024-04-22T08:36:26.402248+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:26.402248+0000","last_undegraded":"2024-04-22T08:36:26.402248+0000","last_fullsized":"2024-04-22T08:36:26.402248+0000","mapping_epoch":38,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887186,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18495,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18698","reported_seq":19455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.367112+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:27.367112+0000","last_peered":"2024-04-22T08:36:27.367112+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:27.367112+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915241,"num_objects":1445,"num_object_clones":0,"num_object_copies":2890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18697,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18340","reported_seq":19016,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.400928+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:27.400928+0000","last_peered":"2024-04-22T08:36:27.400928+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:27.400928+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8300","ondisk_log_start":"37'8300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5775360,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18621","reported_seq":19226,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.377080+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:27.377080+0000","last_peered":"2024-04-22T08:36:27.377080+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:27.377080+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18619,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18638","reported_seq":19355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:27.319976+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:27.319976+0000","last_peered":"2024-04-22T08:36:27.319976+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:27.319976+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5589412,"num_objects":1368,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18635,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":94134125,"num_objects":22994,"num_object_clones":0,"num_object_copies":45988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270710,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15757,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374960128,"data_stored":374892841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117109,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511208,"kb_used_data":168672,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93860632,"statfs":{"total":96636764160,"available":96113287168,"internally_reserved":0,"allocated":172720128,"data_stored":172483706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350744271},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.191},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.36099999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547468,"kb_used_data":70404,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93824372,"statfs":{"total":96636764160,"available":96076156928,"internally_reserved":0,"allocated":72093696,"data_stored":71808568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":488500940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313338,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567340,"kb_used_data":130532,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93804500,"statfs":{"total":96636764160,"available":96055808000,"internally_reserved":0,"allocated":133664768,"data_stored":133438880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":447278803},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.133},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":132317184,"data_stored":132313705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70549504,"data_stored":70493247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172093440,"data_stored":172085889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:34.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:35.584 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:35.656 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-22T08:36:34.395290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1595040,"kb_used_data":350908,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281520480,"statfs":{"total":289910292480,"available":288276971520,"internally_reserved":0,"allocated":359329792,"data_stored":358582354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1273941102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,14,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22384640,"num_objects":-5465,"num_object_clones":0,"num_object_copies":-10930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"41'18875","reported_seq":19597,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.314936+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:32.314936+0000","last_peered":"2024-04-22T08:36:32.314936+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:32.314936+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5296128,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18875,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18872","reported_seq":18992,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.349050+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:31.349050+0000","last_peered":"2024-04-22T08:36:31.349050+0000","last_clean":"2024-04-22T08:36:31.349050+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:31.349050+0000","last_undegraded":"2024-04-22T08:36:31.349050+0000","last_fullsized":"2024-04-22T08:36:31.349050+0000","mapping_epoch":27,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":1360,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18551","reported_seq":19088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.411987+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:31.411987+0000","last_peered":"2024-04-22T08:36:31.411987+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:31.411987+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373976,"num_objects":1313,"num_object_clones":0,"num_object_copies":2626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15378,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18799","reported_seq":18836,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.347434+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:31.347434+0000","last_peered":"2024-04-22T08:36:31.347434+0000","last_clean":"2024-04-22T08:36:31.347434+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:31.347434+0000","last_undegraded":"2024-04-22T08:36:31.347434+0000","last_fullsized":"2024-04-22T08:36:31.347434+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5386240,"num_objects":1315,"num_object_clones":0,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18799,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18634","reported_seq":18779,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.287551+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:32.287551+0000","last_peered":"2024-04-22T08:36:32.287551+0000","last_clean":"2024-04-22T08:36:32.287551+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:32.287551+0000","last_undegraded":"2024-04-22T08:36:32.287551+0000","last_fullsized":"2024-04-22T08:36:32.287551+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5263360,"num_objects":1285,"num_object_clones":0,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15726,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18758","reported_seq":26700,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.012856+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:32.012856+0000","last_peered":"2024-04-22T08:36:32.012856+0000","last_clean":"2024-04-22T08:36:32.012856+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:32.012856+0000","last_undegraded":"2024-04-22T08:36:32.012856+0000","last_fullsized":"2024-04-22T08:36:32.012856+0000","mapping_epoch":40,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5279744,"num_objects":1289,"num_object_clones":0,"num_object_copies":2578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15627,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18591","reported_seq":26333,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.980998+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:31.980998+0000","last_peered":"2024-04-22T08:36:31.980998+0000","last_clean":"2024-04-22T08:36:31.980998+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:31.980998+0000","last_undegraded":"2024-04-22T08:36:31.980998+0000","last_fullsized":"2024-04-22T08:36:31.980998+0000","mapping_epoch":36,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5214208,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14959,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18529","reported_seq":18680,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.381240+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:32.381240+0000","last_peered":"2024-04-22T08:36:32.381240+0000","last_clean":"2024-04-22T08:36:32.381240+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:32.381240+0000","last_undegraded":"2024-04-22T08:36:32.381240+0000","last_fullsized":"2024-04-22T08:36:32.381240+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5230592,"num_objects":1277,"num_object_clones":0,"num_object_copies":2554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15177,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18905","reported_seq":19363,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.419839+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:31.419839+0000","last_peered":"2024-04-22T08:36:31.419839+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:31.419839+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5391570,"num_objects":1318,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18845,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18858","reported_seq":19742,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.373392+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:32.373392+0000","last_peered":"2024-04-22T08:36:32.373392+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:32.373392+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5440722,"num_objects":1330,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18856,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18588","reported_seq":23176,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.483552+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:31.483552+0000","last_peered":"2024-04-22T08:36:31.483552+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:31.483552+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5492736,"num_objects":1341,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15042,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18614","reported_seq":26295,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.419860+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:31.419860+0000","last_peered":"2024-04-22T08:36:31.419860+0000","last_clean":"2024-04-22T08:36:31.419860+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:31.419860+0000","last_undegraded":"2024-04-22T08:36:31.419860+0000","last_fullsized":"2024-04-22T08:36:31.419860+0000","mapping_epoch":38,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5407954,"num_objects":1322,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18612,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18836","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.375949+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:32.375949+0000","last_peered":"2024-04-22T08:36:32.375949+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:32.375949+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5349993,"num_objects":1307,"num_object_clones":0,"num_object_copies":2614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18835,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18480","reported_seq":19156,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.278003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:32.278003+0000","last_peered":"2024-04-22T08:36:32.278003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:32.278003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14711,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18770","reported_seq":19374,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.436600+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:32.436600+0000","last_peered":"2024-04-22T08:36:32.436600+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:32.436600+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194962,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18768,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18768","reported_seq":19485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.304760+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:32.304760+0000","last_peered":"2024-04-22T08:36:32.304760+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:32.304760+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5056932,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18765,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356855808,"data_stored":356788521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117110,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511068,"kb_used_data":164436,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93860772,"statfs":{"total":96636764160,"available":96113430528,"internally_reserved":0,"allocated":168382464,"data_stored":168146042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.442}]}]},{"osd":1,"up_from":33,"seq":141733920872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531552,"kb_used_data":62680,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93840288,"statfs":{"total":96636764160,"available":96092454912,"internally_reserved":0,"allocated":64184320,"data_stored":63899192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.251}]}]},{"osd":0,"up_from":21,"seq":90194313339,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552420,"kb_used_data":123792,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93819420,"statfs":{"total":96636764160,"available":96071086080,"internally_reserved":0,"allocated":126763008,"data_stored":126537120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":2.1299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":2.2160000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":2.2519999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125333504,"data_stored":125330025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63782912,"data_stored":63726655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167739392,"data_stored":167731841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:35.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:36.425 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:36.426 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:36.487 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-04-22T08:36:36.397496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1595040,"kb_used_data":350908,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281520480,"statfs":{"total":289910292480,"available":288276971520,"internally_reserved":0,"allocated":359329792,"data_stored":358582354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1273941102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,14,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-4324,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004614"},"pg_stats":[{"pgid":"2.f","version":"41'18875","reported_seq":19597,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.314936+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:32.314936+0000","last_peered":"2024-04-22T08:36:32.314936+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:32.314936+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5296128,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18875,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18872","reported_seq":18992,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.349050+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:31.349050+0000","last_peered":"2024-04-22T08:36:31.349050+0000","last_clean":"2024-04-22T08:36:31.349050+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:31.349050+0000","last_undegraded":"2024-04-22T08:36:31.349050+0000","last_fullsized":"2024-04-22T08:36:31.349050+0000","mapping_epoch":27,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":1360,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18551","reported_seq":19088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.411987+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:31.411987+0000","last_peered":"2024-04-22T08:36:31.411987+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:31.411987+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373976,"num_objects":1313,"num_object_clones":0,"num_object_copies":2626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15378,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18799","reported_seq":18836,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.347434+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:31.347434+0000","last_peered":"2024-04-22T08:36:31.347434+0000","last_clean":"2024-04-22T08:36:31.347434+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:31.347434+0000","last_undegraded":"2024-04-22T08:36:31.347434+0000","last_fullsized":"2024-04-22T08:36:31.347434+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5386240,"num_objects":1315,"num_object_clones":0,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18799,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18634","reported_seq":18779,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.287551+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:32.287551+0000","last_peered":"2024-04-22T08:36:32.287551+0000","last_clean":"2024-04-22T08:36:32.287551+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:32.287551+0000","last_undegraded":"2024-04-22T08:36:32.287551+0000","last_fullsized":"2024-04-22T08:36:32.287551+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5263360,"num_objects":1285,"num_object_clones":0,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15726,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18758","reported_seq":26700,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.012856+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:32.012856+0000","last_peered":"2024-04-22T08:36:32.012856+0000","last_clean":"2024-04-22T08:36:32.012856+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:32.012856+0000","last_undegraded":"2024-04-22T08:36:32.012856+0000","last_fullsized":"2024-04-22T08:36:32.012856+0000","mapping_epoch":40,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5279744,"num_objects":1289,"num_object_clones":0,"num_object_copies":2578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15627,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18591","reported_seq":26333,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.980998+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:31.980998+0000","last_peered":"2024-04-22T08:36:31.980998+0000","last_clean":"2024-04-22T08:36:31.980998+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:31.980998+0000","last_undegraded":"2024-04-22T08:36:31.980998+0000","last_fullsized":"2024-04-22T08:36:31.980998+0000","mapping_epoch":36,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5214208,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14959,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18529","reported_seq":18680,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.381240+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:32.381240+0000","last_peered":"2024-04-22T08:36:32.381240+0000","last_clean":"2024-04-22T08:36:32.381240+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:32.381240+0000","last_undegraded":"2024-04-22T08:36:32.381240+0000","last_fullsized":"2024-04-22T08:36:32.381240+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5230592,"num_objects":1277,"num_object_clones":0,"num_object_copies":2554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15177,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18905","reported_seq":19363,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.419839+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:31.419839+0000","last_peered":"2024-04-22T08:36:31.419839+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:31.419839+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5391570,"num_objects":1318,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18845,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18858","reported_seq":19742,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.373392+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:32.373392+0000","last_peered":"2024-04-22T08:36:32.373392+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:32.373392+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5440722,"num_objects":1330,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18856,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18588","reported_seq":23176,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.483552+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:31.483552+0000","last_peered":"2024-04-22T08:36:31.483552+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:31.483552+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5492736,"num_objects":1341,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15042,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18614","reported_seq":26295,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.419860+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:31.419860+0000","last_peered":"2024-04-22T08:36:31.419860+0000","last_clean":"2024-04-22T08:36:31.419860+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:31.419860+0000","last_undegraded":"2024-04-22T08:36:31.419860+0000","last_fullsized":"2024-04-22T08:36:31.419860+0000","mapping_epoch":38,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5407954,"num_objects":1322,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18612,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18836","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.375949+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:32.375949+0000","last_peered":"2024-04-22T08:36:32.375949+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:32.375949+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5349993,"num_objects":1307,"num_object_clones":0,"num_object_copies":2614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18835,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18480","reported_seq":19156,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.278003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:32.278003+0000","last_peered":"2024-04-22T08:36:32.278003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:32.278003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14711,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18770","reported_seq":19374,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.436600+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:32.436600+0000","last_peered":"2024-04-22T08:36:32.436600+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:32.436600+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194962,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18768,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18768","reported_seq":19485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.304760+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:32.304760+0000","last_peered":"2024-04-22T08:36:32.304760+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:32.304760+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5056932,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18765,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356855808,"data_stored":356788521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117110,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511068,"kb_used_data":164436,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93860772,"statfs":{"total":96636764160,"available":96113430528,"internally_reserved":0,"allocated":168382464,"data_stored":168146042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.442}]}]},{"osd":1,"up_from":33,"seq":141733920872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531552,"kb_used_data":62680,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93840288,"statfs":{"total":96636764160,"available":96092454912,"internally_reserved":0,"allocated":64184320,"data_stored":63899192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.251}]}]},{"osd":0,"up_from":21,"seq":90194313339,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552420,"kb_used_data":123792,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93819420,"statfs":{"total":96636764160,"available":96071086080,"internally_reserved":0,"allocated":126763008,"data_stored":126537120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":2.1299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":2.2160000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":2.2519999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125333504,"data_stored":125330025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63782912,"data_stored":63726655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167739392,"data_stored":167731841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:36.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:36.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'18875", 'reported_seq': 19597, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.314936+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:32.314936+0000', 'last_peered': '2024-04-22T08:36:32.314936+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:32.314936+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'8800", 'ondisk_log_start': "37'8800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5296128, 'num_objects': 1293, 'num_object_clones': 0, 'num_object_copies': 2586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18875, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2175}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:36.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'18551", 'reported_seq': 19088, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:31.411987+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:31.411987+0000', 'last_peered': '2024-04-22T08:36:31.411987+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:31.411987+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'8500", 'ondisk_log_start': "37'8500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5373976, 'num_objects': 1313, 'num_object_clones': 0, 'num_object_copies': 2626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1313, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15378, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'18905", 'reported_seq': 19363, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:31.419839+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:31.419839+0000', 'last_peered': '2024-04-22T08:36:31.419839+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:31.419839+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5391570, 'num_objects': 1318, 'num_object_clones': 0, 'num_object_copies': 2636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18845, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'18858", 'reported_seq': 19742, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.373392+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:32.373392+0000', 'last_peered': '2024-04-22T08:36:32.373392+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:32.373392+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'8800", 'ondisk_log_start': "37'8800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5440722, 'num_objects': 1330, 'num_object_clones': 0, 'num_object_copies': 2660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18856, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2041}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'18588", 'reported_seq': 23176, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:31.483552+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:31.483552+0000', 'last_peered': '2024-04-22T08:36:31.483552+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:31.483552+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'8500", 'ondisk_log_start': "37'8500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5492736, 'num_objects': 1341, 'num_object_clones': 0, 'num_object_copies': 2682, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1535, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15042, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2217, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -194}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18836", 'reported_seq': 19594, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.375949+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:32.375949+0000', 'last_peered': '2024-04-22T08:36:32.375949+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:32.375949+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'8800", 'ondisk_log_start': "37'8800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5349993, 'num_objects': 1307, 'num_object_clones': 0, 'num_object_copies': 2614, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18835, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'18480", 'reported_seq': 19156, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.278003+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:32.278003+0000', 'last_peered': '2024-04-22T08:36:32.278003+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:32.278003+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'8400", 'ondisk_log_start': "37'8400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5201920, 'num_objects': 1270, 'num_object_clones': 0, 'num_object_copies': 2540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14711, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.489 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:36.490 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18770", 'reported_seq': 19374, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.436600+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:32.436600+0000', 'last_peered': '2024-04-22T08:36:32.436600+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:32.436600+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'8700", 'ondisk_log_start': "37'8700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5194962, 'num_objects': 1270, 'num_object_clones': 0, 'num_object_copies': 2540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18768, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.490 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:36.490 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18768", 'reported_seq': 19485, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:32.304760+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:32.304760+0000', 'last_peered': '2024-04-22T08:36:32.304760+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:32.304760+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'8700", 'ondisk_log_start': "37'8700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5056932, 'num_objects': 1238, 'num_object_clones': 0, 'num_object_copies': 2476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18765, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:36.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:37.573 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:37.573 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29680/41602 objects degraded (71.343%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":632,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20801,"data_bytes":85151597,"bytes_used":1633320960,"bytes_avail":288276971520,"bytes_total":289910292480,"degraded_objects":29680,"degraded_total":41602,"degraded_ratio":0.71342723907504446,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":278,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":360},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:37.624 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:37.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:38.355 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:38.449 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-04-22T08:36:36.397496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1595040,"kb_used_data":350908,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281520480,"statfs":{"total":289910292480,"available":288276971520,"internally_reserved":0,"allocated":359329792,"data_stored":358582354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1273941102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,14,10,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-4324,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004614"},"pg_stats":[{"pgid":"2.f","version":"41'18875","reported_seq":19597,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.314936+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:32.314936+0000","last_peered":"2024-04-22T08:36:32.314936+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:32.314936+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5296128,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18875,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'18872","reported_seq":18992,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.349050+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:31.349050+0000","last_peered":"2024-04-22T08:36:31.349050+0000","last_clean":"2024-04-22T08:36:31.349050+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:31.349050+0000","last_undegraded":"2024-04-22T08:36:31.349050+0000","last_fullsized":"2024-04-22T08:36:31.349050+0000","mapping_epoch":27,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":1360,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18551","reported_seq":19088,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.411987+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:31.411987+0000","last_peered":"2024-04-22T08:36:31.411987+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:31.411987+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373976,"num_objects":1313,"num_object_clones":0,"num_object_copies":2626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15378,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18799","reported_seq":18836,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.347434+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:31.347434+0000","last_peered":"2024-04-22T08:36:31.347434+0000","last_clean":"2024-04-22T08:36:31.347434+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:31.347434+0000","last_undegraded":"2024-04-22T08:36:31.347434+0000","last_fullsized":"2024-04-22T08:36:31.347434+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5386240,"num_objects":1315,"num_object_clones":0,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18799,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18634","reported_seq":18779,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.287551+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:32.287551+0000","last_peered":"2024-04-22T08:36:32.287551+0000","last_clean":"2024-04-22T08:36:32.287551+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:32.287551+0000","last_undegraded":"2024-04-22T08:36:32.287551+0000","last_fullsized":"2024-04-22T08:36:32.287551+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5263360,"num_objects":1285,"num_object_clones":0,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15726,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18758","reported_seq":26700,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.012856+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:32.012856+0000","last_peered":"2024-04-22T08:36:32.012856+0000","last_clean":"2024-04-22T08:36:32.012856+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:32.012856+0000","last_undegraded":"2024-04-22T08:36:32.012856+0000","last_fullsized":"2024-04-22T08:36:32.012856+0000","mapping_epoch":40,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5279744,"num_objects":1289,"num_object_clones":0,"num_object_copies":2578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15627,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18591","reported_seq":26333,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.980998+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:31.980998+0000","last_peered":"2024-04-22T08:36:31.980998+0000","last_clean":"2024-04-22T08:36:31.980998+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:31.980998+0000","last_undegraded":"2024-04-22T08:36:31.980998+0000","last_fullsized":"2024-04-22T08:36:31.980998+0000","mapping_epoch":36,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5214208,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14959,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18529","reported_seq":18680,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:32.381240+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:32.381240+0000","last_peered":"2024-04-22T08:36:32.381240+0000","last_clean":"2024-04-22T08:36:32.381240+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:32.381240+0000","last_undegraded":"2024-04-22T08:36:32.381240+0000","last_fullsized":"2024-04-22T08:36:32.381240+0000","mapping_epoch":27,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5230592,"num_objects":1277,"num_object_clones":0,"num_object_copies":2554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15177,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'18905","reported_seq":19363,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.419839+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:31.419839+0000","last_peered":"2024-04-22T08:36:31.419839+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:31.419839+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5391570,"num_objects":1318,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18845,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'18858","reported_seq":19742,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.373392+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:32.373392+0000","last_peered":"2024-04-22T08:36:32.373392+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:32.373392+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5440722,"num_objects":1330,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18856,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18588","reported_seq":23176,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:31.483552+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:31.483552+0000","last_peered":"2024-04-22T08:36:31.483552+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:31.483552+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8500","ondisk_log_start":"37'8500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5492736,"num_objects":1341,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15042,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18614","reported_seq":26295,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:31.419860+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:31.419860+0000","last_peered":"2024-04-22T08:36:31.419860+0000","last_clean":"2024-04-22T08:36:31.419860+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:31.419860+0000","last_undegraded":"2024-04-22T08:36:31.419860+0000","last_fullsized":"2024-04-22T08:36:31.419860+0000","mapping_epoch":38,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5407954,"num_objects":1322,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18612,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18836","reported_seq":19594,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.375949+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:32.375949+0000","last_peered":"2024-04-22T08:36:32.375949+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:32.375949+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5349993,"num_objects":1307,"num_object_clones":0,"num_object_copies":2614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18835,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18480","reported_seq":19156,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.278003+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:32.278003+0000","last_peered":"2024-04-22T08:36:32.278003+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:32.278003+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8400","ondisk_log_start":"37'8400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14711,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18770","reported_seq":19374,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.436600+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:32.436600+0000","last_peered":"2024-04-22T08:36:32.436600+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:32.436600+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194962,"num_objects":1270,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18768,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-22T08:35:01.583955+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:01.583955+0000","last_peered":"2024-04-22T08:35:01.583955+0000","last_clean":"2024-04-22T08:35:01.583955+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:01.583955+0000","last_undegraded":"2024-04-22T08:35:01.583955+0000","last_fullsized":"2024-04-22T08:35:01.583955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18768","reported_seq":19485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:32.304760+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:32.304760+0000","last_peered":"2024-04-22T08:36:32.304760+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:32.304760+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5056932,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18765,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":85151597,"num_objects":20801,"num_object_clones":0,"num_object_copies":41602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20801,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272903,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15927,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356855808,"data_stored":356788521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117110,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511068,"kb_used_data":164436,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93860772,"statfs":{"total":96636764160,"available":96113430528,"internally_reserved":0,"allocated":168382464,"data_stored":168146042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":354938575},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,9,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.442}]}]},{"osd":1,"up_from":33,"seq":141733920872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531552,"kb_used_data":62680,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93840288,"statfs":{"total":96636764160,"available":96092454912,"internally_reserved":0,"allocated":64184320,"data_stored":63899192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":480112332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.251}]}]},{"osd":0,"up_from":21,"seq":90194313339,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552420,"kb_used_data":123792,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93819420,"statfs":{"total":96636764160,"available":96071086080,"internally_reserved":0,"allocated":126763008,"data_stored":126537120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":2.1299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":2.2160000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":2.2519999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125333504,"data_stored":125330025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63782912,"data_stored":63726655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167739392,"data_stored":167731841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:41.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:42.283 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:42.319 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-04-22T08:36:40.398519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75399021,"num_objects":18420,"num_object_clones":0,"num_object_copies":36840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275284,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16048,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1593916,"kb_used_data":331736,"kb_used_omap":12,"kb_used_meta":1262131,"kb_avail":281521604,"statfs":{"total":289910292480,"available":288278122496,"internally_reserved":0,"allocated":339697664,"data_stored":338950226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1292422254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,10,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18735104,"num_objects":-4574,"num_object_clones":0,"num_object_copies":-9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004732"},"pg_stats":[{"pgid":"2.f","version":"41'19032","reported_seq":19751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.403172+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:37.403172+0000","last_peered":"2024-04-22T08:36:37.403172+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:37.403172+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19032,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2332}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19015","reported_seq":19136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:36.323783+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:36.323783+0000","last_peered":"2024-04-22T08:36:36.323783+0000","last_clean":"2024-04-22T08:36:36.323783+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:36.323783+0000","last_undegraded":"2024-04-22T08:36:36.323783+0000","last_fullsized":"2024-04-22T08:36:36.323783+0000","mapping_epoch":27,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4984832,"num_objects":1217,"num_object_clones":0,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18692","reported_seq":19227,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:36.357290+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:36.357290+0000","last_peered":"2024-04-22T08:36:36.357290+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:36.357290+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4796440,"num_objects":1172,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15519,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'18931","reported_seq":18968,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:36.469607+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:36.469607+0000","last_peered":"2024-04-22T08:36:36.469607+0000","last_clean":"2024-04-22T08:36:36.469607+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:36.469607+0000","last_undegraded":"2024-04-22T08:36:36.469607+0000","last_fullsized":"2024-04-22T08:36:36.469607+0000","mapping_epoch":27,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4845568,"num_objects":1183,"num_object_clones":0,"num_object_copies":2366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18931,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18788","reported_seq":18933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:37.475987+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:37.475987+0000","last_peered":"2024-04-22T08:36:37.475987+0000","last_clean":"2024-04-22T08:36:37.475987+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:37.475987+0000","last_undegraded":"2024-04-22T08:36:37.475987+0000","last_fullsized":"2024-04-22T08:36:37.475987+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4632576,"num_objects":1131,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'18905","reported_seq":26847,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:37.048821+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:37.048821+0000","last_peered":"2024-04-22T08:36:37.048821+0000","last_clean":"2024-04-22T08:36:37.048821+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:37.048821+0000","last_undegraded":"2024-04-22T08:36:37.048821+0000","last_fullsized":"2024-04-22T08:36:37.048821+0000","mapping_epoch":40,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4677632,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15774,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18735","reported_seq":26477,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:36.999976+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:36.999976+0000","last_peered":"2024-04-22T08:36:36.999976+0000","last_clean":"2024-04-22T08:36:36.999976+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:36.999976+0000","last_undegraded":"2024-04-22T08:36:36.999976+0000","last_fullsized":"2024-04-22T08:36:36.999976+0000","mapping_epoch":36,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4624384,"num_objects":1129,"num_object_clones":0,"num_object_copies":2258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18696","reported_seq":18847,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:37.361456+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:37.361456+0000","last_peered":"2024-04-22T08:36:37.361456+0000","last_clean":"2024-04-22T08:36:37.361456+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:37.361456+0000","last_undegraded":"2024-04-22T08:36:37.361456+0000","last_fullsized":"2024-04-22T08:36:37.361456+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4546560,"num_objects":1110,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15344,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19057","reported_seq":19517,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:36.478880+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:36.478880+0000","last_peered":"2024-04-22T08:36:36.478880+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:36.478880+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4768978,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18997,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19021","reported_seq":19905,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.381653+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:37.381653+0000","last_peered":"2024-04-22T08:36:37.381653+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:37.381653+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773074,"num_objects":1167,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18753","reported_seq":23583,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:36.494638+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:36.494638+0000","last_peered":"2024-04-22T08:36:36.494638+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:36.494638+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4816896,"num_objects":1176,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15207,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18765","reported_seq":26446,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:36.408698+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:36.408698+0000","last_peered":"2024-04-22T08:36:36.408698+0000","last_clean":"2024-04-22T08:36:36.408698+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:36.408698+0000","last_undegraded":"2024-04-22T08:36:36.408698+0000","last_fullsized":"2024-04-22T08:36:36.408698+0000","mapping_epoch":38,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789458,"num_objects":1171,"num_object_clones":0,"num_object_copies":2342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18763,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18982","reported_seq":19740,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.402269+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:37.402269+0000","last_peered":"2024-04-22T08:36:37.402269+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:37.402269+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4751977,"num_objects":1161,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18981,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2224}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18625","reported_seq":19301,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.418131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:37.418131+0000","last_peered":"2024-04-22T08:36:37.418131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:37.418131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4608000,"num_objects":1125,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14856,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18898","reported_seq":19503,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.391699+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:37.391699+0000","last_peered":"2024-04-22T08:36:37.391699+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:37.391699+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4670674,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18896,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18914","reported_seq":19631,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.415559+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:37.415559+0000","last_peered":"2024-04-22T08:36:37.415559+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:37.415559+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4458916,"num_objects":1092,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18911,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":75399021,"num_objects":18420,"num_object_clones":0,"num_object_copies":36840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275284,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16048,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337395712,"data_stored":337328425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117111,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516008,"kb_used_data":159520,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93855832,"statfs":{"total":96636764160,"available":96108371968,"internally_reserved":0,"allocated":163348480,"data_stored":163112058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.372}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.94299999999999995}]}]},{"osd":1,"up_from":33,"seq":141733920873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528492,"kb_used_data":55512,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93843348,"statfs":{"total":96636764160,"available":96095588352,"internally_reserved":0,"allocated":56844288,"data_stored":56559160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.251}]}]},{"osd":0,"up_from":21,"seq":90194313340,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549416,"kb_used_data":116704,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93822424,"statfs":{"total":96636764160,"available":96074162176,"internally_reserved":0,"allocated":119504896,"data_stored":119279008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":443084499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":6.2850000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":5.5110000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":5.5439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118095872,"data_stored":118092393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56463360,"data_stored":56407103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162836480,"data_stored":162828929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:42.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:43.164 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:43.165 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:43.225 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-04-22T08:36:42.398862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":70303597,"num_objects":17176,"num_object_clones":0,"num_object_copies":34352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276528,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16176,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1577736,"kb_used_data":319660,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281537784,"statfs":{"total":289910292480,"available":288294690816,"internally_reserved":0,"allocated":327331840,"data_stored":326584402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1288227950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,9,9,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23830528,"num_objects":-5818,"num_object_clones":0,"num_object_copies":-11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004543"},"pg_stats":[{"pgid":"2.f","version":"41'19032","reported_seq":19751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.403172+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:37.403172+0000","last_peered":"2024-04-22T08:36:37.403172+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:37.403172+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19032,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2332}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19190","reported_seq":19309,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.517608+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:41.517608+0000","last_peered":"2024-04-22T08:36:41.517608+0000","last_clean":"2024-04-22T08:36:41.517608+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:41.517608+0000","last_undegraded":"2024-04-22T08:36:41.517608+0000","last_fullsized":"2024-04-22T08:36:41.517608+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4268032,"num_objects":1042,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16246,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18841","reported_seq":19378,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.430670+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:41.430670+0000","last_peered":"2024-04-22T08:36:41.430670+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:41.430670+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4186136,"num_objects":1023,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15668,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2529}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19082","reported_seq":19119,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.439798+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:41.439798+0000","last_peered":"2024-04-22T08:36:41.439798+0000","last_clean":"2024-04-22T08:36:41.439798+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:41.439798+0000","last_undegraded":"2024-04-22T08:36:41.439798+0000","last_fullsized":"2024-04-22T08:36:41.439798+0000","mapping_epoch":27,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19082,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18788","reported_seq":18933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:37.475987+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:37.475987+0000","last_peered":"2024-04-22T08:36:37.475987+0000","last_clean":"2024-04-22T08:36:37.475987+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:37.475987+0000","last_undegraded":"2024-04-22T08:36:37.475987+0000","last_fullsized":"2024-04-22T08:36:37.475987+0000","mapping_epoch":27,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4632576,"num_objects":1131,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19067","reported_seq":27009,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.013830+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:42.013830+0000","last_peered":"2024-04-22T08:36:42.013830+0000","last_clean":"2024-04-22T08:36:42.013830+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:42.013830+0000","last_undegraded":"2024-04-22T08:36:42.013830+0000","last_fullsized":"2024-04-22T08:36:42.013830+0000","mapping_epoch":40,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4014080,"num_objects":980,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15936,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18869","reported_seq":26611,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.036285+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:42.036285+0000","last_peered":"2024-04-22T08:36:42.036285+0000","last_clean":"2024-04-22T08:36:42.036285+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:42.036285+0000","last_undegraded":"2024-04-22T08:36:42.036285+0000","last_fullsized":"2024-04-22T08:36:42.036285+0000","mapping_epoch":36,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4075520,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15237,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18696","reported_seq":18847,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:37.361456+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:37.361456+0000","last_peered":"2024-04-22T08:36:37.361456+0000","last_clean":"2024-04-22T08:36:37.361456+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:37.361456+0000","last_undegraded":"2024-04-22T08:36:37.361456+0000","last_fullsized":"2024-04-22T08:36:37.361456+0000","mapping_epoch":27,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4546560,"num_objects":1110,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15344,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19215","reported_seq":19675,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.468342+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:41.468342+0000","last_peered":"2024-04-22T08:36:41.468342+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:41.468342+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4121810,"num_objects":1008,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19155,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19021","reported_seq":19905,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.381653+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:37.381653+0000","last_peered":"2024-04-22T08:36:37.381653+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:37.381653+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773074,"num_objects":1167,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2204}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18910","reported_seq":23996,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.513586+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:41.513586+0000","last_peered":"2024-04-22T08:36:41.513586+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:41.513586+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4173824,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15364,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-267}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18923","reported_seq":26604,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.510434+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:41.510434+0000","last_peered":"2024-04-22T08:36:41.510434+0000","last_clean":"2024-04-22T08:36:41.510434+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:41.510434+0000","last_undegraded":"2024-04-22T08:36:41.510434+0000","last_fullsized":"2024-04-22T08:36:41.510434+0000","mapping_epoch":38,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142290,"num_objects":1013,"num_object_clones":0,"num_object_copies":2026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'18982","reported_seq":19740,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.402269+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:37.402269+0000","last_peered":"2024-04-22T08:36:37.402269+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:37.402269+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4751977,"num_objects":1161,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18981,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2224}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18625","reported_seq":19301,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.418131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:37.418131+0000","last_peered":"2024-04-22T08:36:37.418131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:37.418131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8600","ondisk_log_start":"37'8600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4608000,"num_objects":1125,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14856,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'18898","reported_seq":19503,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.391699+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:37.391699+0000","last_peered":"2024-04-22T08:36:37.391699+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:37.391699+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4670674,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18896,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'18914","reported_seq":19631,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:37.415559+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:37.415559+0000","last_peered":"2024-04-22T08:36:37.415559+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:37.415559+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4458916,"num_objects":1092,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18911,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":70303597,"num_objects":17176,"num_object_clones":0,"num_object_copies":34352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276528,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16176,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324837376,"data_stored":324770089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117112,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515236,"kb_used_data":154648,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93856604,"statfs":{"total":96636764160,"available":96109162496,"internally_reserved":0,"allocated":158359552,"data_stored":158123130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":1.077},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":1.113}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.317},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":6.4409999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513084,"kb_used_data":48308,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93858756,"statfs":{"total":96636764160,"available":96111366144,"internally_reserved":0,"allocated":49467392,"data_stored":49182264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":475918028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.318}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313340,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549416,"kb_used_data":116704,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93822424,"statfs":{"total":96636764160,"available":96074162176,"internally_reserved":0,"allocated":119504896,"data_stored":119279008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":443084499},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":6.2850000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":5.5110000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":5.5439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118095872,"data_stored":118092393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49049600,"data_stored":48993343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157691904,"data_stored":157684353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'19032", 'reported_seq': 19751, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.403172+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:37.403172+0000', 'last_peered': '2024-04-22T08:36:37.403172+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:37.403172+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'9000", 'ondisk_log_start': "37'9000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4653056, 'num_objects': 1136, 'num_object_clones': 0, 'num_object_copies': 2272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19032, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2332}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'18841", 'reported_seq': 19378, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:41.430670+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:41.430670+0000', 'last_peered': '2024-04-22T08:36:41.430670+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:41.430670+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'8800", 'ondisk_log_start': "37'8800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4186136, 'num_objects': 1023, 'num_object_clones': 0, 'num_object_copies': 2046, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1023, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15668, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:43.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'19215", 'reported_seq': 19675, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:41.468342+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:41.468342+0000', 'last_peered': '2024-04-22T08:36:41.468342+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:41.468342+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'9200", 'ondisk_log_start': "37'9200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4121810, 'num_objects': 1008, 'num_object_clones': 0, 'num_object_copies': 2016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19155, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'19021", 'reported_seq': 19905, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.381653+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:37.381653+0000', 'last_peered': '2024-04-22T08:36:37.381653+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:37.381653+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'9000", 'ondisk_log_start': "37'9000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4773074, 'num_objects': 1167, 'num_object_clones': 0, 'num_object_copies': 2334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19019, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2204}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'18910", 'reported_seq': 23996, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:41.513586+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:41.513586+0000', 'last_peered': '2024-04-22T08:36:41.513586+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:41.513586+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4173824, 'num_objects': 1019, 'num_object_clones': 0, 'num_object_copies': 2038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1286, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15364, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2466, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -267}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'18982", 'reported_seq': 19740, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.402269+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:37.402269+0000', 'last_peered': '2024-04-22T08:36:37.402269+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:37.402269+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4751977, 'num_objects': 1161, 'num_object_clones': 0, 'num_object_copies': 2322, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18981, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2224}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'18625", 'reported_seq': 19301, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.418131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:37.418131+0000', 'last_peered': '2024-04-22T08:36:37.418131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:37.418131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'8600", 'ondisk_log_start': "37'8600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4608000, 'num_objects': 1125, 'num_object_clones': 0, 'num_object_copies': 2250, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14856, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:43.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'18898", 'reported_seq': 19503, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.391699+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:37.391699+0000', 'last_peered': '2024-04-22T08:36:37.391699+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:37.391699+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'8800", 'ondisk_log_start': "37'8800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4670674, 'num_objects': 1142, 'num_object_clones': 0, 'num_object_copies': 2284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18896, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:43.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'18914", 'reported_seq': 19631, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:37.415559+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:37.415559+0000', 'last_peered': '2024-04-22T08:36:37.415559+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:37.415559+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4458916, 'num_objects': 1092, 'num_object_clones': 0, 'num_object_copies': 2184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18911, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:43.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:44.286 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:44.287 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29431/34352 objects degraded (85.675%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":639,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":17176,"data_bytes":70303597,"bytes_used":1615601664,"bytes_avail":288294690816,"bytes_total":289910292480,"degraded_objects":29431,"degraded_total":34352,"degraded_ratio":0.85674778761061943,"recovering_objects_per_sec":34,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":419,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":484},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:44.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:44.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:45.154 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:45.154 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:45.198 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-04-22T08:36:44.399335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65539949,"num_objects":16013,"num_object_clones":0,"num_object_copies":32026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16176,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1562296,"kb_used_data":312412,"kb_used_omap":12,"kb_used_meta":1249843,"kb_avail":281553224,"statfs":{"total":289910292480,"available":288310501376,"internally_reserved":0,"allocated":319909888,"data_stored":319162450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1279839342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,3,4,14,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24264704,"num_objects":-5924,"num_object_clones":0,"num_object_copies":-11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004584"},"pg_stats":[{"pgid":"2.f","version":"41'19187","reported_seq":19909,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.379583+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:42.379583+0000","last_peered":"2024-04-22T08:36:42.379583+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:42.379583+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018176,"num_objects":981,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19187,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19190","reported_seq":19309,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.517608+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:41.517608+0000","last_peered":"2024-04-22T08:36:41.517608+0000","last_clean":"2024-04-22T08:36:41.517608+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:41.517608+0000","last_undegraded":"2024-04-22T08:36:41.517608+0000","last_fullsized":"2024-04-22T08:36:41.517608+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4268032,"num_objects":1042,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16246,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18841","reported_seq":19378,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.430670+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:41.430670+0000","last_peered":"2024-04-22T08:36:41.430670+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:41.430670+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4186136,"num_objects":1023,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15668,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2529}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19082","reported_seq":19119,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.439798+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:41.439798+0000","last_peered":"2024-04-22T08:36:41.439798+0000","last_clean":"2024-04-22T08:36:41.439798+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:41.439798+0000","last_undegraded":"2024-04-22T08:36:41.439798+0000","last_fullsized":"2024-04-22T08:36:41.439798+0000","mapping_epoch":27,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19082,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'18935","reported_seq":19080,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.446638+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:42.446638+0000","last_peered":"2024-04-22T08:36:42.446638+0000","last_clean":"2024-04-22T08:36:42.446638+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:42.446638+0000","last_undegraded":"2024-04-22T08:36:42.446638+0000","last_fullsized":"2024-04-22T08:36:42.446638+0000","mapping_epoch":27,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":984,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16027,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19067","reported_seq":27009,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.013830+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:42.013830+0000","last_peered":"2024-04-22T08:36:42.013830+0000","last_clean":"2024-04-22T08:36:42.013830+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:42.013830+0000","last_undegraded":"2024-04-22T08:36:42.013830+0000","last_fullsized":"2024-04-22T08:36:42.013830+0000","mapping_epoch":40,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4014080,"num_objects":980,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15936,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'18869","reported_seq":26611,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.036285+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:42.036285+0000","last_peered":"2024-04-22T08:36:42.036285+0000","last_clean":"2024-04-22T08:36:42.036285+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:42.036285+0000","last_undegraded":"2024-04-22T08:36:42.036285+0000","last_fullsized":"2024-04-22T08:36:42.036285+0000","mapping_epoch":36,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4075520,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15237,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18842","reported_seq":18993,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:42.372093+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:42.372093+0000","last_peered":"2024-04-22T08:36:42.372093+0000","last_clean":"2024-04-22T08:36:42.372093+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:42.372093+0000","last_undegraded":"2024-04-22T08:36:42.372093+0000","last_fullsized":"2024-04-22T08:36:42.372093+0000","mapping_epoch":27,"log_start":"37'8800","ondisk_log_start":"37'8800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3948544,"num_objects":964,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15490,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19215","reported_seq":19675,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.468342+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:41.468342+0000","last_peered":"2024-04-22T08:36:41.468342+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:41.468342+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4121810,"num_objects":1008,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19155,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19168","reported_seq":20052,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.451113+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:42.451113+0000","last_peered":"2024-04-22T08:36:42.451113+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:42.451113+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4170962,"num_objects":1020,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19166,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'18910","reported_seq":23996,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:41.513586+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:41.513586+0000","last_peered":"2024-04-22T08:36:41.513586+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:41.513586+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4173824,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15364,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-267}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'18923","reported_seq":26604,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:41.510434+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:41.510434+0000","last_peered":"2024-04-22T08:36:41.510434+0000","last_clean":"2024-04-22T08:36:41.510434+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:41.510434+0000","last_undegraded":"2024-04-22T08:36:41.510434+0000","last_fullsized":"2024-04-22T08:36:41.510434+0000","mapping_epoch":38,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142290,"num_objects":1013,"num_object_clones":0,"num_object_copies":2026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19118","reported_seq":19876,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.479200+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:42.479200+0000","last_peered":"2024-04-22T08:36:42.479200+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:42.479200+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194921,"num_objects":1025,"num_object_clones":0,"num_object_copies":2050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19117,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18771","reported_seq":19447,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.478331+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:42.478331+0000","last_peered":"2024-04-22T08:36:42.478331+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:42.478331+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8700","ondisk_log_start":"37'8700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4009984,"num_objects":979,"num_object_clones":0,"num_object_copies":1958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15002,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19053","reported_seq":19657,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.377382+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:42.377382+0000","last_peered":"2024-04-22T08:36:42.377382+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:42.377382+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4035794,"num_objects":987,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19051,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19045","reported_seq":19762,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:42.466934+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:42.466934+0000","last_peered":"2024-04-22T08:36:42.466934+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:42.466934+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3922340,"num_objects":961,"num_object_clones":0,"num_object_copies":1922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19042,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":65539949,"num_objects":16013,"num_object_clones":0,"num_object_copies":32026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16176,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317526016,"data_stored":317458729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117112,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515236,"kb_used_data":154648,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93856604,"statfs":{"total":96636764160,"available":96109162496,"internally_reserved":0,"allocated":158359552,"data_stored":158123130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":1.077},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":1.113}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.317},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":6.4409999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513084,"kb_used_data":48308,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93858756,"statfs":{"total":96636764160,"available":96111366144,"internally_reserved":0,"allocated":49467392,"data_stored":49182264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":475918028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.318}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313341,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533976,"kb_used_data":109456,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93837864,"statfs":{"total":96636764160,"available":96089972736,"internally_reserved":0,"allocated":112082944,"data_stored":111857056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":434695891},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,0,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.501},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":6.9050000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110784512,"data_stored":110781033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49049600,"data_stored":48993343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157691904,"data_stored":157684353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:48.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:48.992 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:49.090 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-04-22T08:36:48.407205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1559876,"kb_used_data":293612,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281555644,"statfs":{"total":289910292480,"available":288312979456,"internally_reserved":0,"allocated":300658688,"data_stored":299911250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1296616558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,14,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29396992,"num_objects":-7177,"num_object_clones":0,"num_object_copies":-14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009708"},"pg_stats":[{"pgid":"2.f","version":"41'19343","reported_seq":20065,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488009+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:47.488009+0000","last_peered":"2024-04-22T08:36:47.488009+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:47.488009+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3379200,"num_objects":825,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19343,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2643}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19339","reported_seq":19461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.552097+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:46.552097+0000","last_peered":"2024-04-22T08:36:46.552097+0000","last_clean":"2024-04-22T08:36:46.552097+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:46.552097+0000","last_undegraded":"2024-04-22T08:36:46.552097+0000","last_fullsized":"2024-04-22T08:36:46.552097+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3657728,"num_objects":893,"num_object_clones":0,"num_object_copies":1786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16395,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18988","reported_seq":19525,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.459212+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:46.459212+0000","last_peered":"2024-04-22T08:36:46.459212+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:46.459212+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3584024,"num_objects":876,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15815,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19229","reported_seq":19266,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.547972+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:46.547972+0000","last_peered":"2024-04-22T08:36:46.547972+0000","last_clean":"2024-04-22T08:36:46.547972+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:46.547972+0000","last_undegraded":"2024-04-22T08:36:46.547972+0000","last_fullsized":"2024-04-22T08:36:46.547972+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3624960,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19229,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19109","reported_seq":19254,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.535529+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:47.535529+0000","last_peered":"2024-04-22T08:36:47.535529+0000","last_clean":"2024-04-22T08:36:47.535529+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:47.535529+0000","last_undegraded":"2024-04-22T08:36:47.535529+0000","last_fullsized":"2024-04-22T08:36:47.535529+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":810,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16201,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19218","reported_seq":27160,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.210752+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:47.210752+0000","last_peered":"2024-04-22T08:36:47.210752+0000","last_clean":"2024-04-22T08:36:47.210752+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:47.210752+0000","last_undegraded":"2024-04-22T08:36:47.210752+0000","last_fullsized":"2024-04-22T08:36:47.210752+0000","mapping_epoch":40,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3395584,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19023","reported_seq":26765,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.193587+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:47.193587+0000","last_peered":"2024-04-22T08:36:47.193587+0000","last_clean":"2024-04-22T08:36:47.193587+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:47.193587+0000","last_undegraded":"2024-04-22T08:36:47.193587+0000","last_fullsized":"2024-04-22T08:36:47.193587+0000","mapping_epoch":36,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3444736,"num_objects":841,"num_object_clones":0,"num_object_copies":1682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15391,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18987","reported_seq":19138,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.471789+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:47.471789+0000","last_peered":"2024-04-22T08:36:47.471789+0000","last_clean":"2024-04-22T08:36:47.471789+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:47.471789+0000","last_undegraded":"2024-04-22T08:36:47.471789+0000","last_fullsized":"2024-04-22T08:36:47.471789+0000","mapping_epoch":27,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3354624,"num_objects":819,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15635,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19360","reported_seq":19820,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.533054+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:46.533054+0000","last_peered":"2024-04-22T08:36:46.533054+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:46.533054+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3527890,"num_objects":863,"num_object_clones":0,"num_object_copies":1726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19300,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2860}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19335","reported_seq":20218,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.505927+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:47.505927+0000","last_peered":"2024-04-22T08:36:47.505927+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:47.505927+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486930,"num_objects":853,"num_object_clones":0,"num_object_copies":1706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19067","reported_seq":24419,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.543320+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:46.543320+0000","last_peered":"2024-04-22T08:36:46.543320+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:46.543320+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3530752,"num_objects":862,"num_object_clones":0,"num_object_copies":1724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15521,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19066","reported_seq":26747,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.511111+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:46.511111+0000","last_peered":"2024-04-22T08:36:46.511111+0000","last_clean":"2024-04-22T08:36:46.511111+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:46.511111+0000","last_undegraded":"2024-04-22T08:36:46.511111+0000","last_fullsized":"2024-04-22T08:36:46.511111+0000","mapping_epoch":38,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3556562,"num_objects":870,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19064,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19269","reported_seq":20027,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488840+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:47.488840+0000","last_peered":"2024-04-22T08:36:47.488840+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:47.488840+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576425,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18901","reported_seq":19577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.504997+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:47.504997+0000","last_peered":"2024-04-22T08:36:47.504997+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:47.504997+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3477504,"num_objects":849,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19194","reported_seq":19799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.332565+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:47.332565+0000","last_peered":"2024-04-22T08:36:47.332565+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:47.332565+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3458258,"num_objects":846,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19192,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19177","reported_seq":19893,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.519621+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:47.519621+0000","last_peered":"2024-04-22T08:36:47.519621+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:47.519621+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3381668,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298037248,"data_stored":297969961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117113,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514568,"kb_used_data":149888,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93857272,"statfs":{"total":96636764160,"available":96109846528,"internally_reserved":0,"allocated":153485312,"data_stored":153248890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373419727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.38}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.252}]}]},{"osd":1,"up_from":33,"seq":141733920875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514272,"kb_used_data":41304,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93857568,"statfs":{"total":96636764160,"available":96110149632,"internally_reserved":0,"allocated":42295296,"data_stored":42010168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313342,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531036,"kb_used_data":102420,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93840804,"statfs":{"total":96636764160,"available":96092983296,"internally_reserved":0,"allocated":104878080,"data_stored":104652192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":6.1740000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":11.141999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103432809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41795584,"data_stored":41739327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152805376,"data_stored":152797825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:49.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:49.963 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:50.011 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-04-22T08:36:48.407205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1559876,"kb_used_data":293612,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281555644,"statfs":{"total":289910292480,"available":288312979456,"internally_reserved":0,"allocated":300658688,"data_stored":299911250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1296616558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,14,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29396992,"num_objects":-7177,"num_object_clones":0,"num_object_copies":-14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009708"},"pg_stats":[{"pgid":"2.f","version":"41'19343","reported_seq":20065,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488009+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:47.488009+0000","last_peered":"2024-04-22T08:36:47.488009+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:47.488009+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3379200,"num_objects":825,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19343,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2643}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19339","reported_seq":19461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.552097+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:46.552097+0000","last_peered":"2024-04-22T08:36:46.552097+0000","last_clean":"2024-04-22T08:36:46.552097+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:46.552097+0000","last_undegraded":"2024-04-22T08:36:46.552097+0000","last_fullsized":"2024-04-22T08:36:46.552097+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3657728,"num_objects":893,"num_object_clones":0,"num_object_copies":1786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16395,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18988","reported_seq":19525,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.459212+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:46.459212+0000","last_peered":"2024-04-22T08:36:46.459212+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:46.459212+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3584024,"num_objects":876,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15815,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19229","reported_seq":19266,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.547972+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:46.547972+0000","last_peered":"2024-04-22T08:36:46.547972+0000","last_clean":"2024-04-22T08:36:46.547972+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:46.547972+0000","last_undegraded":"2024-04-22T08:36:46.547972+0000","last_fullsized":"2024-04-22T08:36:46.547972+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3624960,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19229,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19109","reported_seq":19254,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.535529+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:47.535529+0000","last_peered":"2024-04-22T08:36:47.535529+0000","last_clean":"2024-04-22T08:36:47.535529+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:47.535529+0000","last_undegraded":"2024-04-22T08:36:47.535529+0000","last_fullsized":"2024-04-22T08:36:47.535529+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":810,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16201,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19218","reported_seq":27160,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.210752+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:47.210752+0000","last_peered":"2024-04-22T08:36:47.210752+0000","last_clean":"2024-04-22T08:36:47.210752+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:47.210752+0000","last_undegraded":"2024-04-22T08:36:47.210752+0000","last_fullsized":"2024-04-22T08:36:47.210752+0000","mapping_epoch":40,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3395584,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19023","reported_seq":26765,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.193587+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:47.193587+0000","last_peered":"2024-04-22T08:36:47.193587+0000","last_clean":"2024-04-22T08:36:47.193587+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:47.193587+0000","last_undegraded":"2024-04-22T08:36:47.193587+0000","last_fullsized":"2024-04-22T08:36:47.193587+0000","mapping_epoch":36,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3444736,"num_objects":841,"num_object_clones":0,"num_object_copies":1682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15391,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18987","reported_seq":19138,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.471789+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:47.471789+0000","last_peered":"2024-04-22T08:36:47.471789+0000","last_clean":"2024-04-22T08:36:47.471789+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:47.471789+0000","last_undegraded":"2024-04-22T08:36:47.471789+0000","last_fullsized":"2024-04-22T08:36:47.471789+0000","mapping_epoch":27,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3354624,"num_objects":819,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15635,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19360","reported_seq":19820,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.533054+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:46.533054+0000","last_peered":"2024-04-22T08:36:46.533054+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:46.533054+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3527890,"num_objects":863,"num_object_clones":0,"num_object_copies":1726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19300,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2860}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19335","reported_seq":20218,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.505927+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:47.505927+0000","last_peered":"2024-04-22T08:36:47.505927+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:47.505927+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486930,"num_objects":853,"num_object_clones":0,"num_object_copies":1706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19067","reported_seq":24419,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.543320+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:46.543320+0000","last_peered":"2024-04-22T08:36:46.543320+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:46.543320+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3530752,"num_objects":862,"num_object_clones":0,"num_object_copies":1724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15521,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19066","reported_seq":26747,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.511111+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:46.511111+0000","last_peered":"2024-04-22T08:36:46.511111+0000","last_clean":"2024-04-22T08:36:46.511111+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:46.511111+0000","last_undegraded":"2024-04-22T08:36:46.511111+0000","last_fullsized":"2024-04-22T08:36:46.511111+0000","mapping_epoch":38,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3556562,"num_objects":870,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19064,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19269","reported_seq":20027,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488840+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:47.488840+0000","last_peered":"2024-04-22T08:36:47.488840+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:47.488840+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576425,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18901","reported_seq":19577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.504997+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:47.504997+0000","last_peered":"2024-04-22T08:36:47.504997+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:47.504997+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3477504,"num_objects":849,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19194","reported_seq":19799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.332565+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:47.332565+0000","last_peered":"2024-04-22T08:36:47.332565+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:47.332565+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3458258,"num_objects":846,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19192,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19177","reported_seq":19893,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.519621+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:47.519621+0000","last_peered":"2024-04-22T08:36:47.519621+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:47.519621+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3381668,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298037248,"data_stored":297969961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117113,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514568,"kb_used_data":149888,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93857272,"statfs":{"total":96636764160,"available":96109846528,"internally_reserved":0,"allocated":153485312,"data_stored":153248890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373419727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.38}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.252}]}]},{"osd":1,"up_from":33,"seq":141733920875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514272,"kb_used_data":41304,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93857568,"statfs":{"total":96636764160,"available":96110149632,"internally_reserved":0,"allocated":42295296,"data_stored":42010168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313342,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531036,"kb_used_data":102420,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93840804,"statfs":{"total":96636764160,"available":96092983296,"internally_reserved":0,"allocated":104878080,"data_stored":104652192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":6.1740000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":11.141999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103432809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41795584,"data_stored":41739327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152805376,"data_stored":152797825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'19343", 'reported_seq': 20065, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.488009+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:47.488009+0000', 'last_peered': '2024-04-22T08:36:47.488009+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:47.488009+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'9300", 'ondisk_log_start': "37'9300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3379200, 'num_objects': 825, 'num_object_clones': 0, 'num_object_copies': 1650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19343, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2643}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'18988", 'reported_seq': 19525, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:46.459212+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:46.459212+0000', 'last_peered': '2024-04-22T08:36:46.459212+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:46.459212+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3584024, 'num_objects': 876, 'num_object_clones': 0, 'num_object_copies': 1752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 876, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15815, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'19360", 'reported_seq': 19820, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:46.533054+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:46.533054+0000', 'last_peered': '2024-04-22T08:36:46.533054+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:46.533054+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'9300", 'ondisk_log_start': "37'9300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3527890, 'num_objects': 863, 'num_object_clones': 0, 'num_object_copies': 1726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19300, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:50.013 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'19335", 'reported_seq': 20218, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.505927+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:47.505927+0000', 'last_peered': '2024-04-22T08:36:47.505927+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:47.505927+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'9300", 'ondisk_log_start': "37'9300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3486930, 'num_objects': 853, 'num_object_clones': 0, 'num_object_copies': 1706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19333, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2518}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19067", 'reported_seq': 24419, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:46.543320+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:46.543320+0000', 'last_peered': '2024-04-22T08:36:46.543320+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:46.543320+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9000", 'ondisk_log_start': "37'9000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3530752, 'num_objects': 862, 'num_object_clones': 0, 'num_object_copies': 1724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15521, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2599, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -291}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'19269", 'reported_seq': 20027, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.488840+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:47.488840+0000', 'last_peered': '2024-04-22T08:36:47.488840+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:47.488840+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'9200", 'ondisk_log_start': "37'9200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3576425, 'num_objects': 874, 'num_object_clones': 0, 'num_object_copies': 1748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19268, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'18901", 'reported_seq': 19577, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.504997+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:47.504997+0000', 'last_peered': '2024-04-22T08:36:47.504997+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:47.504997+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'8900", 'ondisk_log_start': "37'8900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3477504, 'num_objects': 849, 'num_object_clones': 0, 'num_object_copies': 1698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15132, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:50.014 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'19194", 'reported_seq': 19799, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.332565+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:47.332565+0000', 'last_peered': '2024-04-22T08:36:47.332565+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:47.332565+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'9100", 'ondisk_log_start': "37'9100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3458258, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 1692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19192, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.015 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:50.015 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'19177", 'reported_seq': 19893, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:47.519621+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:47.519621+0000', 'last_peered': '2024-04-22T08:36:47.519621+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:47.519621+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'9100", 'ondisk_log_start': "37'9100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3381668, 'num_objects': 829, 'num_object_clones': 0, 'num_object_copies': 1658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19174, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:50.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:51.266 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:51.266 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29298/27248 objects degraded (107.523%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":646,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13624,"data_bytes":55754605,"bytes_used":1597313024,"bytes_avail":288312979456,"bytes_total":289910292480,"degraded_objects":29298,"degraded_total":27248,"degraded_ratio":1.0752348796241926,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":261,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":399},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-22T08:36:18.391592+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-22T08:36:51.292 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:51.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:51.976 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:52.017 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-04-22T08:36:50.407644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1559876,"kb_used_data":293612,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281555644,"statfs":{"total":289910292480,"available":288312979456,"internally_reserved":0,"allocated":300658688,"data_stored":299911250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1296616558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,14,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19644416,"num_objects":-4796,"num_object_clones":0,"num_object_copies":-9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4796,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009692"},"pg_stats":[{"pgid":"2.f","version":"41'19343","reported_seq":20065,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488009+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:47.488009+0000","last_peered":"2024-04-22T08:36:47.488009+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:47.488009+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3379200,"num_objects":825,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19343,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2643}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19339","reported_seq":19461,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.552097+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:46.552097+0000","last_peered":"2024-04-22T08:36:46.552097+0000","last_clean":"2024-04-22T08:36:46.552097+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:46.552097+0000","last_undegraded":"2024-04-22T08:36:46.552097+0000","last_fullsized":"2024-04-22T08:36:46.552097+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3657728,"num_objects":893,"num_object_clones":0,"num_object_copies":1786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16395,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'18988","reported_seq":19525,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.459212+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:46.459212+0000","last_peered":"2024-04-22T08:36:46.459212+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:46.459212+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3584024,"num_objects":876,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15815,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19229","reported_seq":19266,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.547972+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:46.547972+0000","last_peered":"2024-04-22T08:36:46.547972+0000","last_clean":"2024-04-22T08:36:46.547972+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:46.547972+0000","last_undegraded":"2024-04-22T08:36:46.547972+0000","last_fullsized":"2024-04-22T08:36:46.547972+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3624960,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19229,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19109","reported_seq":19254,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.535529+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:47.535529+0000","last_peered":"2024-04-22T08:36:47.535529+0000","last_clean":"2024-04-22T08:36:47.535529+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:47.535529+0000","last_undegraded":"2024-04-22T08:36:47.535529+0000","last_fullsized":"2024-04-22T08:36:47.535529+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3317760,"num_objects":810,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16201,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19218","reported_seq":27160,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.210752+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:47.210752+0000","last_peered":"2024-04-22T08:36:47.210752+0000","last_clean":"2024-04-22T08:36:47.210752+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:47.210752+0000","last_undegraded":"2024-04-22T08:36:47.210752+0000","last_fullsized":"2024-04-22T08:36:47.210752+0000","mapping_epoch":40,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3395584,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19023","reported_seq":26765,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.193587+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:47.193587+0000","last_peered":"2024-04-22T08:36:47.193587+0000","last_clean":"2024-04-22T08:36:47.193587+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:47.193587+0000","last_undegraded":"2024-04-22T08:36:47.193587+0000","last_fullsized":"2024-04-22T08:36:47.193587+0000","mapping_epoch":36,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3444736,"num_objects":841,"num_object_clones":0,"num_object_copies":1682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15391,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'18987","reported_seq":19138,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:47.471789+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:47.471789+0000","last_peered":"2024-04-22T08:36:47.471789+0000","last_clean":"2024-04-22T08:36:47.471789+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:47.471789+0000","last_undegraded":"2024-04-22T08:36:47.471789+0000","last_fullsized":"2024-04-22T08:36:47.471789+0000","mapping_epoch":27,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3354624,"num_objects":819,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15635,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19360","reported_seq":19820,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.533054+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:46.533054+0000","last_peered":"2024-04-22T08:36:46.533054+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:46.533054+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3527890,"num_objects":863,"num_object_clones":0,"num_object_copies":1726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19300,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2860}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19335","reported_seq":20218,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.505927+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:47.505927+0000","last_peered":"2024-04-22T08:36:47.505927+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:47.505927+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486930,"num_objects":853,"num_object_clones":0,"num_object_copies":1706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19067","reported_seq":24419,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:46.543320+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:46.543320+0000","last_peered":"2024-04-22T08:36:46.543320+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:46.543320+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3530752,"num_objects":862,"num_object_clones":0,"num_object_copies":1724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15521,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19066","reported_seq":26747,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:46.511111+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:46.511111+0000","last_peered":"2024-04-22T08:36:46.511111+0000","last_clean":"2024-04-22T08:36:46.511111+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:46.511111+0000","last_undegraded":"2024-04-22T08:36:46.511111+0000","last_fullsized":"2024-04-22T08:36:46.511111+0000","mapping_epoch":38,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3556562,"num_objects":870,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19064,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19269","reported_seq":20027,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.488840+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:47.488840+0000","last_peered":"2024-04-22T08:36:47.488840+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:47.488840+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576425,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'18901","reported_seq":19577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.504997+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:47.504997+0000","last_peered":"2024-04-22T08:36:47.504997+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:47.504997+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'8900","ondisk_log_start":"37'8900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3477504,"num_objects":849,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2741}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19194","reported_seq":19799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.332565+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:47.332565+0000","last_peered":"2024-04-22T08:36:47.332565+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:47.332565+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3458258,"num_objects":846,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19192,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19177","reported_seq":19893,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:47.519621+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:47.519621+0000","last_peered":"2024-04-22T08:36:47.519621+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:47.519621+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3381668,"num_objects":829,"num_object_clones":0,"num_object_copies":1658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19174,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":55754605,"num_objects":13624,"num_object_clones":0,"num_object_copies":27248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280080,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16309,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298037248,"data_stored":297969961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117113,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514568,"kb_used_data":149888,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93857272,"statfs":{"total":96636764160,"available":96109846528,"internally_reserved":0,"allocated":153485312,"data_stored":153248890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373419727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.38}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.252}]}]},{"osd":1,"up_from":33,"seq":141733920875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514272,"kb_used_data":41304,"kb_used_omap":4,"kb_used_meta":472955,"kb_avail":93857568,"statfs":{"total":96636764160,"available":96110149632,"internally_reserved":0,"allocated":42295296,"data_stored":42010168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":484306636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313342,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531036,"kb_used_data":102420,"kb_used_omap":4,"kb_used_meta":428603,"kb_avail":93840804,"statfs":{"total":96636764160,"available":96092983296,"internally_reserved":0,"allocated":104878080,"data_stored":104652192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":438890195},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":6.1740000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":11.141999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103436288,"data_stored":103432809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41795584,"data_stored":41739327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152805376,"data_stored":152797825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:55.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:55.812 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:55.813 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:55.860 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-04-22T08:36:54.408542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46628717,"num_objects":11396,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282308,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16475,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1566216,"kb_used_data":276128,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281549304,"statfs":{"total":289910292480,"available":288306487296,"internally_reserved":0,"allocated":282755072,"data_stored":282007634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,10,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23674880,"num_objects":-5780,"num_object_clones":0,"num_object_copies":-11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009680"},"pg_stats":[{"pgid":"2.f","version":"41'19477","reported_seq":20199,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.541711+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:52.541711+0000","last_peered":"2024-04-22T08:36:52.541711+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:52.541711+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2830336,"num_objects":691,"num_object_clones":0,"num_object_copies":1382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19477,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2777}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19489","reported_seq":19611,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.392863+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:51.392863+0000","last_peered":"2024-04-22T08:36:51.392863+0000","last_clean":"2024-04-22T08:36:51.392863+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:51.392863+0000","last_undegraded":"2024-04-22T08:36:51.392863+0000","last_fullsized":"2024-04-22T08:36:51.392863+0000","mapping_epoch":27,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3043328,"num_objects":743,"num_object_clones":0,"num_object_copies":1486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16545,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19140","reported_seq":19677,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.565130+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:51.565130+0000","last_peered":"2024-04-22T08:36:51.565130+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:51.565130+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2961432,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15967,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19374","reported_seq":19411,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.351986+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:51.351986+0000","last_peered":"2024-04-22T08:36:51.351986+0000","last_clean":"2024-04-22T08:36:51.351986+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:51.351986+0000","last_undegraded":"2024-04-22T08:36:51.351986+0000","last_fullsized":"2024-04-22T08:36:51.351986+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3031040,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19374,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19235","reported_seq":19380,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.506793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:52.506793+0000","last_peered":"2024-04-22T08:36:52.506793+0000","last_clean":"2024-04-22T08:36:52.506793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:52.506793+0000","last_undegraded":"2024-04-22T08:36:52.506793+0000","last_fullsized":"2024-04-22T08:36:52.506793+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2801664,"num_objects":684,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16327,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19352","reported_seq":27294,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.264050+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:52.264050+0000","last_peered":"2024-04-22T08:36:52.264050+0000","last_clean":"2024-04-22T08:36:52.264050+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:52.264050+0000","last_undegraded":"2024-04-22T08:36:52.264050+0000","last_fullsized":"2024-04-22T08:36:52.264050+0000","mapping_epoch":40,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2846720,"num_objects":695,"num_object_clones":0,"num_object_copies":1390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19160","reported_seq":26903,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.219778+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:52.219778+0000","last_peered":"2024-04-22T08:36:52.219778+0000","last_clean":"2024-04-22T08:36:52.219778+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:52.219778+0000","last_undegraded":"2024-04-22T08:36:52.219778+0000","last_fullsized":"2024-04-22T08:36:52.219778+0000","mapping_epoch":36,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":704,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15528,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19136","reported_seq":19287,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.416241+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:52.416241+0000","last_peered":"2024-04-22T08:36:52.416241+0000","last_clean":"2024-04-22T08:36:52.416241+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:52.416241+0000","last_undegraded":"2024-04-22T08:36:52.416241+0000","last_fullsized":"2024-04-22T08:36:52.416241+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2744320,"num_objects":670,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19492","reported_seq":19952,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.539513+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:51.539513+0000","last_peered":"2024-04-22T08:36:51.539513+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:51.539513+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2987218,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19432,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19463","reported_seq":20347,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.478680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:52.478680+0000","last_peered":"2024-04-22T08:36:52.478680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:52.478680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2962642,"num_objects":725,"num_object_clones":0,"num_object_copies":1450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2646}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19205","reported_seq":24889,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.574674+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:51.574674+0000","last_peered":"2024-04-22T08:36:51.574674+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:51.574674+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15659,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19196","reported_seq":26876,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.499331+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:51.499331+0000","last_peered":"2024-04-22T08:36:51.499331+0000","last_clean":"2024-04-22T08:36:51.499331+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:51.499331+0000","last_undegraded":"2024-04-22T08:36:51.499331+0000","last_fullsized":"2024-04-22T08:36:51.499331+0000","mapping_epoch":38,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3024082,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19194,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19413","reported_seq":20170,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.570692+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:52.570692+0000","last_peered":"2024-04-22T08:36:52.570692+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:52.570692+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2986601,"num_objects":730,"num_object_clones":0,"num_object_copies":1460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19039","reported_seq":19714,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.568947+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:52.568947+0000","last_peered":"2024-04-22T08:36:52.568947+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:52.568947+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2912256,"num_objects":711,"num_object_clones":0,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15270,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19334","reported_seq":19939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.461573+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:52.461573+0000","last_peered":"2024-04-22T08:36:52.461573+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:52.461573+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2884818,"num_objects":706,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19332,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19328","reported_seq":20045,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.531890+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:52.531890+0000","last_peered":"2024-04-22T08:36:52.531890+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:52.531890+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2763172,"num_objects":678,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19325,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2920}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":46628717,"num_objects":11396,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282308,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16475,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279793664,"data_stored":279726377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117114,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497764,"kb_used_data":145372,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93874076,"statfs":{"total":96636764160,"available":96127053824,"internally_reserved":0,"allocated":148860928,"data_stored":148624506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.27100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920876,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517804,"kb_used_data":34980,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93854036,"statfs":{"total":96636764160,"available":96106532864,"internally_reserved":0,"allocated":35819520,"data_stored":35534392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":4.2439999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.314},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313343,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550648,"kb_used_data":95776,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93821192,"statfs":{"total":96636764160,"available":96072900608,"internally_reserved":0,"allocated":98074624,"data_stored":97848736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":3.6549999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96595968,"data_stored":96592489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34996224,"data_stored":34939967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148201472,"data_stored":148193921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:55.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:56.661 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:56.716 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-22T08:36:56.409024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46628717,"num_objects":11396,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282308,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16475,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1566216,"kb_used_data":276128,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281549304,"statfs":{"total":289910292480,"available":288306487296,"internally_reserved":0,"allocated":282755072,"data_stored":282007634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,10,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18911232,"num_objects":-4617,"num_object_clones":0,"num_object_copies":-9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009689"},"pg_stats":[{"pgid":"2.f","version":"41'19477","reported_seq":20199,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.541711+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:52.541711+0000","last_peered":"2024-04-22T08:36:52.541711+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:52.541711+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2830336,"num_objects":691,"num_object_clones":0,"num_object_copies":1382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19477,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2777}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19489","reported_seq":19611,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.392863+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:51.392863+0000","last_peered":"2024-04-22T08:36:51.392863+0000","last_clean":"2024-04-22T08:36:51.392863+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:51.392863+0000","last_undegraded":"2024-04-22T08:36:51.392863+0000","last_fullsized":"2024-04-22T08:36:51.392863+0000","mapping_epoch":27,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3043328,"num_objects":743,"num_object_clones":0,"num_object_copies":1486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16545,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19140","reported_seq":19677,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.565130+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:51.565130+0000","last_peered":"2024-04-22T08:36:51.565130+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:51.565130+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2961432,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15967,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19374","reported_seq":19411,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.351986+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:51.351986+0000","last_peered":"2024-04-22T08:36:51.351986+0000","last_clean":"2024-04-22T08:36:51.351986+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:51.351986+0000","last_undegraded":"2024-04-22T08:36:51.351986+0000","last_fullsized":"2024-04-22T08:36:51.351986+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3031040,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19374,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19235","reported_seq":19380,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.506793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:52.506793+0000","last_peered":"2024-04-22T08:36:52.506793+0000","last_clean":"2024-04-22T08:36:52.506793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:52.506793+0000","last_undegraded":"2024-04-22T08:36:52.506793+0000","last_fullsized":"2024-04-22T08:36:52.506793+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2801664,"num_objects":684,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16327,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19352","reported_seq":27294,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.264050+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:52.264050+0000","last_peered":"2024-04-22T08:36:52.264050+0000","last_clean":"2024-04-22T08:36:52.264050+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:52.264050+0000","last_undegraded":"2024-04-22T08:36:52.264050+0000","last_fullsized":"2024-04-22T08:36:52.264050+0000","mapping_epoch":40,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2846720,"num_objects":695,"num_object_clones":0,"num_object_copies":1390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19160","reported_seq":26903,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.219778+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:52.219778+0000","last_peered":"2024-04-22T08:36:52.219778+0000","last_clean":"2024-04-22T08:36:52.219778+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:52.219778+0000","last_undegraded":"2024-04-22T08:36:52.219778+0000","last_fullsized":"2024-04-22T08:36:52.219778+0000","mapping_epoch":36,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":704,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15528,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19136","reported_seq":19287,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:52.416241+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:52.416241+0000","last_peered":"2024-04-22T08:36:52.416241+0000","last_clean":"2024-04-22T08:36:52.416241+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:52.416241+0000","last_undegraded":"2024-04-22T08:36:52.416241+0000","last_fullsized":"2024-04-22T08:36:52.416241+0000","mapping_epoch":27,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2744320,"num_objects":670,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19492","reported_seq":19952,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.539513+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:51.539513+0000","last_peered":"2024-04-22T08:36:51.539513+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:51.539513+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2987218,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19432,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19463","reported_seq":20347,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.478680+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:52.478680+0000","last_peered":"2024-04-22T08:36:52.478680+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:52.478680+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2962642,"num_objects":725,"num_object_clones":0,"num_object_copies":1450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2646}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19205","reported_seq":24889,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:51.574674+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:51.574674+0000","last_peered":"2024-04-22T08:36:51.574674+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:51.574674+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15659,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19196","reported_seq":26876,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:51.499331+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:51.499331+0000","last_peered":"2024-04-22T08:36:51.499331+0000","last_clean":"2024-04-22T08:36:51.499331+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:51.499331+0000","last_undegraded":"2024-04-22T08:36:51.499331+0000","last_fullsized":"2024-04-22T08:36:51.499331+0000","mapping_epoch":38,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3024082,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19194,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19413","reported_seq":20170,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.570692+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:52.570692+0000","last_peered":"2024-04-22T08:36:52.570692+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:52.570692+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2986601,"num_objects":730,"num_object_clones":0,"num_object_copies":1460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19039","reported_seq":19714,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.568947+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:52.568947+0000","last_peered":"2024-04-22T08:36:52.568947+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:52.568947+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9000","ondisk_log_start":"37'9000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2912256,"num_objects":711,"num_object_clones":0,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15270,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19334","reported_seq":19939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.461573+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:52.461573+0000","last_peered":"2024-04-22T08:36:52.461573+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:52.461573+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2884818,"num_objects":706,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19332,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19328","reported_seq":20045,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:52.531890+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:52.531890+0000","last_peered":"2024-04-22T08:36:52.531890+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:52.531890+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2763172,"num_objects":678,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19325,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2920}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":46628717,"num_objects":11396,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282308,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16475,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279793664,"data_stored":279726377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117114,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497764,"kb_used_data":145372,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93874076,"statfs":{"total":96636764160,"available":96127053824,"internally_reserved":0,"allocated":148860928,"data_stored":148624506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.27100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920876,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517804,"kb_used_data":34980,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93854036,"statfs":{"total":96636764160,"available":96106532864,"internally_reserved":0,"allocated":35819520,"data_stored":35534392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":4.2439999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.314},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313343,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550648,"kb_used_data":95776,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93821192,"statfs":{"total":96636764160,"available":96072900608,"internally_reserved":0,"allocated":98074624,"data_stored":97848736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":3.6549999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96595968,"data_stored":96592489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34996224,"data_stored":34939967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148201472,"data_stored":148193921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:36:56.717 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'19477", 'reported_seq': 20199, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.541711+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:52.541711+0000', 'last_peered': '2024-04-22T08:36:52.541711+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:52.541711+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2830336, 'num_objects': 691, 'num_object_clones': 0, 'num_object_copies': 1382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19477, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2777}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19140", 'reported_seq': 19677, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:51.565130+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:36:51.565130+0000', 'last_peered': '2024-04-22T08:36:51.565130+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:36:51.565130+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9100", 'ondisk_log_start': "37'9100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2961432, 'num_objects': 724, 'num_object_clones': 0, 'num_object_copies': 1448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 724, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15967, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'19492", 'reported_seq': 19952, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:51.539513+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:36:51.539513+0000', 'last_peered': '2024-04-22T08:36:51.539513+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:36:51.539513+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2987218, 'num_objects': 731, 'num_object_clones': 0, 'num_object_copies': 1462, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19432, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'19463", 'reported_seq': 20347, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.478680+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:52.478680+0000', 'last_peered': '2024-04-22T08:36:52.478680+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:52.478680+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2962642, 'num_objects': 725, 'num_object_clones': 0, 'num_object_copies': 1450, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2646}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:36:56.718 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19205", 'reported_seq': 24889, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:51.574674+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:36:51.574674+0000', 'last_peered': '2024-04-22T08:36:51.574674+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:36:51.574674+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9200", 'ondisk_log_start': "37'9200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2965504, 'num_objects': 724, 'num_object_clones': 0, 'num_object_copies': 1448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 987, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15659, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2765, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'19413", 'reported_seq': 20170, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.570692+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:52.570692+0000', 'last_peered': '2024-04-22T08:36:52.570692+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:52.570692+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2986601, 'num_objects': 730, 'num_object_clones': 0, 'num_object_copies': 1460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19412, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19039", 'reported_seq': 19714, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.568947+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:52.568947+0000', 'last_peered': '2024-04-22T08:36:52.568947+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:52.568947+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9000", 'ondisk_log_start': "37'9000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2912256, 'num_objects': 711, 'num_object_clones': 0, 'num_object_copies': 1422, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15270, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'19334", 'reported_seq': 19939, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.461573+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:52.461573+0000', 'last_peered': '2024-04-22T08:36:52.461573+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:52.461573+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'9300", 'ondisk_log_start': "37'9300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2884818, 'num_objects': 706, 'num_object_clones': 0, 'num_object_copies': 1412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19332, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:36:56.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'19328", 'reported_seq': 20045, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:52.531890+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:52.531890+0000', 'last_peered': '2024-04-22T08:36:52.531890+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:52.531890+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'9300", 'ondisk_log_start': "37'9300", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2763172, 'num_objects': 678, 'num_object_clones': 0, 'num_object_copies': 1356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19325, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:36:56.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:36:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:57.612 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29132/22792 objects degraded (127.817%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":652,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":11396,"data_bytes":46628717,"bytes_used":1603805184,"bytes_avail":288306487296,"bytes_total":289910292480,"degraded_objects":29132,"degraded_total":22792,"degraded_ratio":1.2781677781677783,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":299,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":384},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:36:57.673 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:36:57.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:36:58.534 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:36:58.536 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:36:58.586 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-04-22T08:36:58.409503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36937581,"num_objects":9030,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9030,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284674,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1547208,"kb_used_data":257116,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281568312,"statfs":{"total":289910292480,"available":288325951488,"internally_reserved":0,"allocated":263286784,"data_stored":262539346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,2,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28602368,"num_objects":-6983,"num_object_clones":0,"num_object_copies":-13966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6983,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009692"},"pg_stats":[{"pgid":"2.f","version":"41'19623","reported_seq":20345,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.519010+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:57.519010+0000","last_peered":"2024-04-22T08:36:57.519010+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:57.519010+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2232320,"num_objects":545,"num_object_clones":0,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19623,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19628","reported_seq":19750,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:56.411098+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:36:56.411098+0000","last_peered":"2024-04-22T08:36:56.411098+0000","last_clean":"2024-04-22T08:36:56.411098+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:36:56.411098+0000","last_undegraded":"2024-04-22T08:36:56.411098+0000","last_fullsized":"2024-04-22T08:36:56.411098+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16684,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19288","reported_seq":19825,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:56.574650+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:36:56.574650+0000","last_peered":"2024-04-22T08:36:56.574650+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:36:56.574650+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2355224,"num_objects":576,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16115,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19537","reported_seq":19574,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:56.513808+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:36:56.513808+0000","last_peered":"2024-04-22T08:36:56.513808+0000","last_clean":"2024-04-22T08:36:56.513808+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:36:56.513808+0000","last_undegraded":"2024-04-22T08:36:56.513808+0000","last_fullsized":"2024-04-22T08:36:56.513808+0000","mapping_epoch":27,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2363392,"num_objects":577,"num_object_clones":0,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19537,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19399","reported_seq":19542,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.448180+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:57.448180+0000","last_peered":"2024-04-22T08:36:57.448180+0000","last_clean":"2024-04-22T08:36:57.448180+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:57.448180+0000","last_undegraded":"2024-04-22T08:36:57.448180+0000","last_fullsized":"2024-04-22T08:36:57.448180+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2129920,"num_objects":520,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16491,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19510","reported_seq":27452,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.299918+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:36:57.299918+0000","last_peered":"2024-04-22T08:36:57.299918+0000","last_clean":"2024-04-22T08:36:57.299918+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:36:57.299918+0000","last_undegraded":"2024-04-22T08:36:57.299918+0000","last_fullsized":"2024-04-22T08:36:57.299918+0000","mapping_epoch":40,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2199552,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16379,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19313","reported_seq":27056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.240025+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:36:57.240025+0000","last_peered":"2024-04-22T08:36:57.240025+0000","last_clean":"2024-04-22T08:36:57.240025+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:36:57.240025+0000","last_undegraded":"2024-04-22T08:36:57.240025+0000","last_fullsized":"2024-04-22T08:36:57.240025+0000","mapping_epoch":36,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2256896,"num_objects":551,"num_object_clones":0,"num_object_copies":1102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15681,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19257","reported_seq":19408,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.459493+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:57.459493+0000","last_peered":"2024-04-22T08:36:57.459493+0000","last_clean":"2024-04-22T08:36:57.459493+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:57.459493+0000","last_undegraded":"2024-04-22T08:36:57.459493+0000","last_fullsized":"2024-04-22T08:36:57.459493+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2248704,"num_objects":549,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15905,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19651","reported_seq":20111,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:56.513271+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:36:56.513271+0000","last_peered":"2024-04-22T08:36:56.513271+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:36:56.513271+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2335954,"num_objects":572,"num_object_clones":0,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19591,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3151}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19610","reported_seq":20494,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.556511+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:57.556511+0000","last_peered":"2024-04-22T08:36:57.556511+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:57.556511+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2360530,"num_objects":578,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19608,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19353","reported_seq":25301,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:56.593910+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:36:56.593910+0000","last_peered":"2024-04-22T08:36:56.593910+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:36:56.593910+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":576,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15807,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2897,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19347","reported_seq":27028,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:56.568241+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:36:56.568241+0000","last_peered":"2024-04-22T08:36:56.568241+0000","last_clean":"2024-04-22T08:36:56.568241+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:36:56.568241+0000","last_undegraded":"2024-04-22T08:36:56.568241+0000","last_fullsized":"2024-04-22T08:36:56.568241+0000","mapping_epoch":38,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2405586,"num_objects":589,"num_object_clones":0,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19345,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19552","reported_seq":20310,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.569319+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:57.569319+0000","last_peered":"2024-04-22T08:36:57.569319+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:57.569319+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2417257,"num_objects":591,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19551,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19188","reported_seq":19864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.535610+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:57.535610+0000","last_peered":"2024-04-22T08:36:57.535610+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:57.535610+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2301952,"num_objects":562,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15419,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19476","reported_seq":20078,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.568542+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:57.568542+0000","last_peered":"2024-04-22T08:36:57.568542+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:57.568542+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2303186,"num_objects":564,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19474,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19467","reported_seq":20184,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.530334+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:57.530334+0000","last_peered":"2024-04-22T08:36:57.530334+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:57.530334+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2193828,"num_objects":539,"num_object_clones":0,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19464,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":36937581,"num_objects":9030,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9030,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284674,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260263936,"data_stored":260196649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117115,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497092,"kb_used_data":140588,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93874748,"statfs":{"total":96636764160,"available":96127741952,"internally_reserved":0,"allocated":143962112,"data_stored":143725690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.318},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.23400000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518800,"kb_used_data":27780,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93853040,"statfs":{"total":96636764160,"available":96105512960,"internally_reserved":0,"allocated":28446720,"data_stored":28161592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.221},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.373}]}]},{"osd":0,"up_from":21,"seq":90194313344,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531316,"kb_used_data":88748,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93840524,"statfs":{"total":96636764160,"available":96092696576,"internally_reserved":0,"allocated":90877952,"data_stored":90652064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":453177043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":10.435},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":10.890000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":10.487}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89448448,"data_stored":89444969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27561984,"data_stored":27505727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143253504,"data_stored":143245953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:01.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:02.864 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:02.866 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:03.273 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-22T08:37:02.410495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31854445,"num_objects":7789,"num_object_clones":0,"num_object_copies":15578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285915,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161040,"ondisk_log_size":161040,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1526912,"kb_used_data":245032,"kb_used_omap":12,"kb_used_meta":1281843,"kb_avail":281588608,"statfs":{"total":289910292480,"available":288346734592,"internally_reserved":0,"allocated":250912768,"data_stored":250165330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1312607342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,8,7,6,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23900160,"num_objects":-5835,"num_object_clones":0,"num_object_copies":-11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"2.f","version":"41'19623","reported_seq":20345,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.519010+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:57.519010+0000","last_peered":"2024-04-22T08:36:57.519010+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:57.519010+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2232320,"num_objects":545,"num_object_clones":0,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19623,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19776","reported_seq":19898,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.603551+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:01.603551+0000","last_peered":"2024-04-22T08:37:01.603551+0000","last_clean":"2024-04-22T08:37:01.603551+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:01.603551+0000","last_undegraded":"2024-04-22T08:37:01.603551+0000","last_fullsized":"2024-04-22T08:37:01.603551+0000","mapping_epoch":27,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":456,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16832,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19456","reported_seq":19993,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.601475+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:01.601475+0000","last_peered":"2024-04-22T08:37:01.601475+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:01.601475+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1667096,"num_objects":408,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16283,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3144}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19694","reported_seq":19731,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.529279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:01.529279+0000","last_peered":"2024-04-22T08:37:01.529279+0000","last_clean":"2024-04-22T08:37:01.529279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:01.529279+0000","last_undegraded":"2024-04-22T08:37:01.529279+0000","last_fullsized":"2024-04-22T08:37:01.529279+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1720320,"num_objects":420,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19694,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19399","reported_seq":19542,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.448180+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:57.448180+0000","last_peered":"2024-04-22T08:36:57.448180+0000","last_clean":"2024-04-22T08:36:57.448180+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:57.448180+0000","last_undegraded":"2024-04-22T08:36:57.448180+0000","last_fullsized":"2024-04-22T08:36:57.448180+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2129920,"num_objects":520,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16491,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19656","reported_seq":27598,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.291622+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:02.291622+0000","last_peered":"2024-04-22T08:37:02.291622+0000","last_clean":"2024-04-22T08:37:02.291622+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:02.291622+0000","last_undegraded":"2024-04-22T08:37:02.291622+0000","last_fullsized":"2024-04-22T08:37:02.291622+0000","mapping_epoch":40,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1601536,"num_objects":391,"num_object_clones":0,"num_object_copies":782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16525,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19481","reported_seq":27224,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.379618+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:02.379618+0000","last_peered":"2024-04-22T08:37:02.379618+0000","last_clean":"2024-04-22T08:37:02.379618+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:02.379618+0000","last_undegraded":"2024-04-22T08:37:02.379618+0000","last_fullsized":"2024-04-22T08:37:02.379618+0000","mapping_epoch":36,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15849,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19257","reported_seq":19408,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.459493+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:57.459493+0000","last_peered":"2024-04-22T08:36:57.459493+0000","last_clean":"2024-04-22T08:36:57.459493+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:57.459493+0000","last_undegraded":"2024-04-22T08:36:57.459493+0000","last_fullsized":"2024-04-22T08:36:57.459493+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2248704,"num_objects":549,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15905,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19806","reported_seq":20266,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.572928+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:01.572928+0000","last_peered":"2024-04-22T08:37:01.572928+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:01.572928+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701074,"num_objects":417,"num_object_clones":0,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19746,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19610","reported_seq":20494,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.556511+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:57.556511+0000","last_peered":"2024-04-22T08:36:57.556511+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:57.556511+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2360530,"num_objects":578,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19608,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19500","reported_seq":25686,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.564089+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:01.564089+0000","last_peered":"2024-04-22T08:37:01.564089+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:01.564089+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1757184,"num_objects":429,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15954,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19499","reported_seq":27180,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.575085+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:01.575085+0000","last_peered":"2024-04-22T08:37:01.575085+0000","last_clean":"2024-04-22T08:37:01.575085+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:01.575085+0000","last_undegraded":"2024-04-22T08:37:01.575085+0000","last_fullsized":"2024-04-22T08:37:01.575085+0000","mapping_epoch":38,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1782994,"num_objects":437,"num_object_clones":0,"num_object_copies":874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19497,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19552","reported_seq":20310,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.569319+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:57.569319+0000","last_peered":"2024-04-22T08:36:57.569319+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:57.569319+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2417257,"num_objects":591,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19551,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19188","reported_seq":19864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.535610+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:57.535610+0000","last_peered":"2024-04-22T08:36:57.535610+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:57.535610+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2301952,"num_objects":562,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15419,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19476","reported_seq":20078,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.568542+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:57.568542+0000","last_peered":"2024-04-22T08:36:57.568542+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:57.568542+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2303186,"num_objects":564,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19474,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19467","reported_seq":20184,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.530334+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:57.530334+0000","last_peered":"2024-04-22T08:36:57.530334+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:57.530334+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2193828,"num_objects":539,"num_object_clones":0,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19464,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":31854445,"num_objects":7789,"num_object_clones":0,"num_object_copies":15578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285915,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":247791616,"data_stored":247724329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161040,"ondisk_log_size":161040,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117116,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496316,"kb_used_data":135732,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93875524,"statfs":{"total":96636764160,"available":96128536576,"internally_reserved":0,"allocated":138989568,"data_stored":138753146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.419},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.29299999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920878,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499280,"kb_used_data":20552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93872560,"statfs":{"total":96636764160,"available":96125501440,"internally_reserved":0,"allocated":21045248,"data_stored":20760120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":6.0629999999999997},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.35699999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313344,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531316,"kb_used_data":88748,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93840524,"statfs":{"total":96636764160,"available":96092696576,"internally_reserved":0,"allocated":90877952,"data_stored":90652064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":453177043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":10.435},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":10.890000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":10.487}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89448448,"data_stored":89444969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20033536,"data_stored":19977279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138309632,"data_stored":138302081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:03.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:04.183 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:04.186 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:04.241 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-22T08:37:02.410495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31854445,"num_objects":7789,"num_object_clones":0,"num_object_copies":15578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285915,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161040,"ondisk_log_size":161040,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1526912,"kb_used_data":245032,"kb_used_omap":12,"kb_used_meta":1281843,"kb_avail":281588608,"statfs":{"total":289910292480,"available":288346734592,"internally_reserved":0,"allocated":250912768,"data_stored":250165330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1312607342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,8,7,6,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23900160,"num_objects":-5835,"num_object_clones":0,"num_object_copies":-11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5835,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"2.f","version":"41'19623","reported_seq":20345,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.519010+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:36:57.519010+0000","last_peered":"2024-04-22T08:36:57.519010+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:36:57.519010+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2232320,"num_objects":545,"num_object_clones":0,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19623,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19776","reported_seq":19898,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.603551+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:01.603551+0000","last_peered":"2024-04-22T08:37:01.603551+0000","last_clean":"2024-04-22T08:37:01.603551+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:01.603551+0000","last_undegraded":"2024-04-22T08:37:01.603551+0000","last_fullsized":"2024-04-22T08:37:01.603551+0000","mapping_epoch":27,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":456,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16832,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19456","reported_seq":19993,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.601475+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:01.601475+0000","last_peered":"2024-04-22T08:37:01.601475+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:01.601475+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1667096,"num_objects":408,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16283,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3144}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19694","reported_seq":19731,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.529279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:01.529279+0000","last_peered":"2024-04-22T08:37:01.529279+0000","last_clean":"2024-04-22T08:37:01.529279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:01.529279+0000","last_undegraded":"2024-04-22T08:37:01.529279+0000","last_fullsized":"2024-04-22T08:37:01.529279+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1720320,"num_objects":420,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19694,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19399","reported_seq":19542,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.448180+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:36:57.448180+0000","last_peered":"2024-04-22T08:36:57.448180+0000","last_clean":"2024-04-22T08:36:57.448180+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:36:57.448180+0000","last_undegraded":"2024-04-22T08:36:57.448180+0000","last_fullsized":"2024-04-22T08:36:57.448180+0000","mapping_epoch":27,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2129920,"num_objects":520,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16491,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19656","reported_seq":27598,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.291622+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:02.291622+0000","last_peered":"2024-04-22T08:37:02.291622+0000","last_clean":"2024-04-22T08:37:02.291622+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:02.291622+0000","last_undegraded":"2024-04-22T08:37:02.291622+0000","last_fullsized":"2024-04-22T08:37:02.291622+0000","mapping_epoch":40,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1601536,"num_objects":391,"num_object_clones":0,"num_object_copies":782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16525,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19481","reported_seq":27224,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.379618+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:02.379618+0000","last_peered":"2024-04-22T08:37:02.379618+0000","last_clean":"2024-04-22T08:37:02.379618+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:02.379618+0000","last_undegraded":"2024-04-22T08:37:02.379618+0000","last_fullsized":"2024-04-22T08:37:02.379618+0000","mapping_epoch":36,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15849,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19257","reported_seq":19408,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:36:57.459493+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:36:57.459493+0000","last_peered":"2024-04-22T08:36:57.459493+0000","last_clean":"2024-04-22T08:36:57.459493+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:36:57.459493+0000","last_undegraded":"2024-04-22T08:36:57.459493+0000","last_fullsized":"2024-04-22T08:36:57.459493+0000","mapping_epoch":27,"log_start":"37'9200","ondisk_log_start":"37'9200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2248704,"num_objects":549,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15905,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19806","reported_seq":20266,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.572928+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:01.572928+0000","last_peered":"2024-04-22T08:37:01.572928+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:01.572928+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701074,"num_objects":417,"num_object_clones":0,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19746,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19610","reported_seq":20494,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.556511+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:36:57.556511+0000","last_peered":"2024-04-22T08:36:57.556511+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:36:57.556511+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2360530,"num_objects":578,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19608,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2793}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19500","reported_seq":25686,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.564089+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:01.564089+0000","last_peered":"2024-04-22T08:37:01.564089+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:01.564089+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1757184,"num_objects":429,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15954,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19499","reported_seq":27180,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.575085+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:01.575085+0000","last_peered":"2024-04-22T08:37:01.575085+0000","last_clean":"2024-04-22T08:37:01.575085+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:01.575085+0000","last_undegraded":"2024-04-22T08:37:01.575085+0000","last_fullsized":"2024-04-22T08:37:01.575085+0000","mapping_epoch":38,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1782994,"num_objects":437,"num_object_clones":0,"num_object_copies":874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19497,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19552","reported_seq":20310,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.569319+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:36:57.569319+0000","last_peered":"2024-04-22T08:36:57.569319+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:36:57.569319+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2417257,"num_objects":591,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19551,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19188","reported_seq":19864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.535610+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:36:57.535610+0000","last_peered":"2024-04-22T08:36:57.535610+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:36:57.535610+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9100","ondisk_log_start":"37'9100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2301952,"num_objects":562,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15419,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19476","reported_seq":20078,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.568542+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:36:57.568542+0000","last_peered":"2024-04-22T08:36:57.568542+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:36:57.568542+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2303186,"num_objects":564,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19474,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19467","reported_seq":20184,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:36:57.530334+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:36:57.530334+0000","last_peered":"2024-04-22T08:36:57.530334+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:36:57.530334+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2193828,"num_objects":539,"num_object_clones":0,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19464,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":31854445,"num_objects":7789,"num_object_clones":0,"num_object_copies":15578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285915,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":247791616,"data_stored":247724329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161040,"ondisk_log_size":161040,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117116,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496316,"kb_used_data":135732,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93875524,"statfs":{"total":96636764160,"available":96128536576,"internally_reserved":0,"allocated":138989568,"data_stored":138753146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.419},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.29299999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920878,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499280,"kb_used_data":20552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93872560,"statfs":{"total":96636764160,"available":96125501440,"internally_reserved":0,"allocated":21045248,"data_stored":20760120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":6.0629999999999997},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.35699999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313344,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531316,"kb_used_data":88748,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93840524,"statfs":{"total":96636764160,"available":96092696576,"internally_reserved":0,"allocated":90877952,"data_stored":90652064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":453177043},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":10.435},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":10.890000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":10.487}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89448448,"data_stored":89444969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20033536,"data_stored":19977279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138309632,"data_stored":138302081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'19623", 'reported_seq': 20345, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.519010+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:36:57.519010+0000', 'last_peered': '2024-04-22T08:36:57.519010+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:36:57.519010+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'9600", 'ondisk_log_start': "37'9600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2232320, 'num_objects': 545, 'num_object_clones': 0, 'num_object_copies': 1090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19623, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2923}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19456", 'reported_seq': 19993, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:01.601475+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:01.601475+0000', 'last_peered': '2024-04-22T08:37:01.601475+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:01.601475+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1667096, 'num_objects': 408, 'num_object_clones': 0, 'num_object_copies': 816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 408, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16283, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3144}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:04.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'19806", 'reported_seq': 20266, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:01.572928+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:01.572928+0000', 'last_peered': '2024-04-22T08:37:01.572928+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:01.572928+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1701074, 'num_objects': 417, 'num_object_clones': 0, 'num_object_copies': 834, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19746, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3306}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'19610", 'reported_seq': 20494, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.556511+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:36:57.556511+0000', 'last_peered': '2024-04-22T08:36:57.556511+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:36:57.556511+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'9600", 'ondisk_log_start': "37'9600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2360530, 'num_objects': 578, 'num_object_clones': 0, 'num_object_copies': 1156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19608, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2793}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19500", 'reported_seq': 25686, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:01.564089+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:37:01.564089+0000', 'last_peered': '2024-04-22T08:37:01.564089+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:37:01.564089+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1757184, 'num_objects': 429, 'num_object_clones': 0, 'num_object_copies': 858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 736, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15954, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3016, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -307}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'19552", 'reported_seq': 20310, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.569319+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:36:57.569319+0000', 'last_peered': '2024-04-22T08:36:57.569319+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:36:57.569319+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'9500", 'ondisk_log_start': "37'9500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2417257, 'num_objects': 591, 'num_object_clones': 0, 'num_object_copies': 1182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19551, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19188", 'reported_seq': 19864, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.535610+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:36:57.535610+0000', 'last_peered': '2024-04-22T08:36:57.535610+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:36:57.535610+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9100", 'ondisk_log_start': "37'9100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2301952, 'num_objects': 562, 'num_object_clones': 0, 'num_object_copies': 1124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15419, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'19476", 'reported_seq': 20078, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.568542+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:36:57.568542+0000', 'last_peered': '2024-04-22T08:36:57.568542+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:36:57.568542+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2303186, 'num_objects': 564, 'num_object_clones': 0, 'num_object_copies': 1128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19474, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.244 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:04.245 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'19467", 'reported_seq': 20184, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:36:57.530334+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:36:57.530334+0000', 'last_peered': '2024-04-22T08:36:57.530334+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:36:57.530334+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2193828, 'num_objects': 539, 'num_object_clones': 0, 'num_object_copies': 1078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19464, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:04.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:05.464 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:05.464 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28881/13214 objects degraded (218.564%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":660,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":6607,"data_bytes":27012973,"bytes_used":1564762112,"bytes_avail":288345530368,"bytes_total":289910292480,"degraded_objects":28881,"degraded_total":13214,"degraded_ratio":2.1856364461934312,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":251,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":491},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:05.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:05.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:06.351 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:06.351 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:06.403 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-22T08:37:04.410967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27012973,"num_objects":6607,"num_object_clones":0,"num_object_copies":13214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287097,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1528088,"kb_used_data":238008,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281587432,"statfs":{"total":289910292480,"available":288345530368,"internally_reserved":0,"allocated":243720192,"data_stored":242972754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,10,10,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24162304,"num_objects":-5899,"num_object_clones":0,"num_object_copies":-11798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5899,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002904"},"pg_stats":[{"pgid":"2.f","version":"41'19761","reported_seq":20483,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.553546+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:02.553546+0000","last_peered":"2024-04-22T08:37:02.553546+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:02.553546+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1667072,"num_objects":407,"num_object_clones":0,"num_object_copies":814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19761,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3061}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19776","reported_seq":19898,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.603551+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:01.603551+0000","last_peered":"2024-04-22T08:37:01.603551+0000","last_clean":"2024-04-22T08:37:01.603551+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:01.603551+0000","last_undegraded":"2024-04-22T08:37:01.603551+0000","last_fullsized":"2024-04-22T08:37:01.603551+0000","mapping_epoch":27,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1867776,"num_objects":456,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16832,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19456","reported_seq":19993,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.601475+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:01.601475+0000","last_peered":"2024-04-22T08:37:01.601475+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:01.601475+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1667096,"num_objects":408,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16283,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3144}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19694","reported_seq":19731,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.529279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:01.529279+0000","last_peered":"2024-04-22T08:37:01.529279+0000","last_clean":"2024-04-22T08:37:01.529279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:01.529279+0000","last_undegraded":"2024-04-22T08:37:01.529279+0000","last_fullsized":"2024-04-22T08:37:01.529279+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1720320,"num_objects":420,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19694,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19544","reported_seq":19689,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.527607+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:02.527607+0000","last_peered":"2024-04-22T08:37:02.527607+0000","last_clean":"2024-04-22T08:37:02.527607+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:02.527607+0000","last_undegraded":"2024-04-22T08:37:02.527607+0000","last_fullsized":"2024-04-22T08:37:02.527607+0000","mapping_epoch":27,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536000,"num_objects":375,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16636,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19656","reported_seq":27598,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.291622+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:02.291622+0000","last_peered":"2024-04-22T08:37:02.291622+0000","last_clean":"2024-04-22T08:37:02.291622+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:02.291622+0000","last_undegraded":"2024-04-22T08:37:02.291622+0000","last_fullsized":"2024-04-22T08:37:02.291622+0000","mapping_epoch":40,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1601536,"num_objects":391,"num_object_clones":0,"num_object_copies":782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16525,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19481","reported_seq":27224,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.379618+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:02.379618+0000","last_peered":"2024-04-22T08:37:02.379618+0000","last_clean":"2024-04-22T08:37:02.379618+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:02.379618+0000","last_undegraded":"2024-04-22T08:37:02.379618+0000","last_fullsized":"2024-04-22T08:37:02.379618+0000","mapping_epoch":36,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15849,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19409","reported_seq":19560,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:02.588344+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:02.588344+0000","last_peered":"2024-04-22T08:37:02.588344+0000","last_clean":"2024-04-22T08:37:02.588344+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:02.588344+0000","last_undegraded":"2024-04-22T08:37:02.588344+0000","last_fullsized":"2024-04-22T08:37:02.588344+0000","mapping_epoch":27,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626112,"num_objects":397,"num_object_clones":0,"num_object_copies":794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16057,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19806","reported_seq":20266,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.572928+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:01.572928+0000","last_peered":"2024-04-22T08:37:01.572928+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:01.572928+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701074,"num_objects":417,"num_object_clones":0,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19746,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3306}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19769","reported_seq":20653,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.573689+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:02.573689+0000","last_peered":"2024-04-22T08:37:02.573689+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:02.573689+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1709266,"num_objects":419,"num_object_clones":0,"num_object_copies":838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19767,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2952}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19500","reported_seq":25686,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:01.564089+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:01.564089+0000","last_peered":"2024-04-22T08:37:01.564089+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:01.564089+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1757184,"num_objects":429,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15954,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19499","reported_seq":27180,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:01.575085+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:01.575085+0000","last_peered":"2024-04-22T08:37:01.575085+0000","last_clean":"2024-04-22T08:37:01.575085+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:01.575085+0000","last_undegraded":"2024-04-22T08:37:01.575085+0000","last_fullsized":"2024-04-22T08:37:01.575085+0000","mapping_epoch":38,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1782994,"num_objects":437,"num_object_clones":0,"num_object_copies":874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19497,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19701","reported_seq":20459,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.554344+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:02.554344+0000","last_peered":"2024-04-22T08:37:02.554344+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:02.554344+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1806953,"num_objects":442,"num_object_clones":0,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19700,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19334","reported_seq":20010,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.574569+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:02.574569+0000","last_peered":"2024-04-22T08:37:02.574569+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:02.574569+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9300","ondisk_log_start":"37'9300","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":416,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15565,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19609","reported_seq":20214,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.544827+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:02.544827+0000","last_peered":"2024-04-22T08:37:02.544827+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:02.544827+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1758418,"num_objects":431,"num_object_clones":0,"num_object_copies":862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19607,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19627","reported_seq":20344,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:02.536070+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:02.536070+0000","last_peered":"2024-04-22T08:37:02.536070+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:02.536070+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1538468,"num_objects":379,"num_object_clones":0,"num_object_copies":758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19624,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":27012973,"num_objects":6607,"num_object_clones":0,"num_object_copies":13214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287097,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240398336,"data_stored":240331049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117116,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496316,"kb_used_data":135732,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93875524,"statfs":{"total":96636764160,"available":96128536576,"internally_reserved":0,"allocated":138989568,"data_stored":138753146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.56799999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":1.419},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.29299999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920878,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499280,"kb_used_data":20552,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93872560,"statfs":{"total":96636764160,"available":96125501440,"internally_reserved":0,"allocated":21045248,"data_stored":20760120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":6.0629999999999997},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.35699999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313345,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532492,"kb_used_data":81724,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93839348,"statfs":{"total":96636764160,"available":96091492352,"internally_reserved":0,"allocated":83685376,"data_stored":83459488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":5.6600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":5.702}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82055168,"data_stored":82051689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20033536,"data_stored":19977279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138309632,"data_stored":138302081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:09.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:10.185 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:10.292 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-04-22T08:37:08.411857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17575789,"num_objects":4303,"num_object_clones":0,"num_object_copies":8606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289401,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16880,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1525872,"kb_used_data":219396,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281589648,"statfs":{"total":289910292480,"available":288347799552,"internally_reserved":0,"allocated":224661504,"data_stored":223914066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1337773166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29052928,"num_objects":-7093,"num_object_clones":0,"num_object_copies":-14186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"2.f","version":"41'19914","reported_seq":20636,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.476616+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:07.476616+0000","last_peered":"2024-04-22T08:37:07.476616+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:07.476616+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1040384,"num_objects":254,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19914,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3214}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19931","reported_seq":20053,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.456465+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:06.456465+0000","last_peered":"2024-04-22T08:37:06.456465+0000","last_clean":"2024-04-22T08:37:06.456465+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:06.456465+0000","last_undegraded":"2024-04-22T08:37:06.456465+0000","last_fullsized":"2024-04-22T08:37:06.456465+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232896,"num_objects":301,"num_object_clones":0,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16987,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19586","reported_seq":20123,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.499564+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:06.499564+0000","last_peered":"2024-04-22T08:37:06.499564+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:06.499564+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1134616,"num_objects":278,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16413,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19833","reported_seq":19870,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.589231+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:06.589231+0000","last_peered":"2024-04-22T08:37:06.589231+0000","last_clean":"2024-04-22T08:37:06.589231+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:06.589231+0000","last_undegraded":"2024-04-22T08:37:06.589231+0000","last_fullsized":"2024-04-22T08:37:06.589231+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1150976,"num_objects":281,"num_object_clones":0,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19833,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19686","reported_seq":19828,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.496477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:07.496477+0000","last_peered":"2024-04-22T08:37:07.496477+0000","last_clean":"2024-04-22T08:37:07.496477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:07.496477+0000","last_undegraded":"2024-04-22T08:37:07.496477+0000","last_fullsized":"2024-04-22T08:37:07.496477+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":954368,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16778,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19795","reported_seq":27736,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.448202+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:07.448202+0000","last_peered":"2024-04-22T08:37:07.448202+0000","last_clean":"2024-04-22T08:37:07.448202+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:07.448202+0000","last_undegraded":"2024-04-22T08:37:07.448202+0000","last_fullsized":"2024-04-22T08:37:07.448202+0000","mapping_epoch":40,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16664,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19623","reported_seq":27366,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.274855+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:07.274855+0000","last_peered":"2024-04-22T08:37:07.274855+0000","last_clean":"2024-04-22T08:37:07.274855+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:07.274855+0000","last_undegraded":"2024-04-22T08:37:07.274855+0000","last_fullsized":"2024-04-22T08:37:07.274855+0000","mapping_epoch":36,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":987136,"num_objects":241,"num_object_clones":0,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15991,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19546","reported_seq":19697,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.524244+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:07.524244+0000","last_peered":"2024-04-22T08:37:07.524244+0000","last_clean":"2024-04-22T08:37:07.524244+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:07.524244+0000","last_undegraded":"2024-04-22T08:37:07.524244+0000","last_fullsized":"2024-04-22T08:37:07.524244+0000","mapping_epoch":27,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1064960,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16194,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19953","reported_seq":20413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.482380+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:06.482380+0000","last_peered":"2024-04-22T08:37:06.482380+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:06.482380+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1098962,"num_objects":270,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19893,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19913","reported_seq":20797,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.577367+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:07.577367+0000","last_peered":"2024-04-22T08:37:07.577367+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:07.577367+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119442,"num_objects":275,"num_object_clones":0,"num_object_copies":550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19911,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19637","reported_seq":26131,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.597753+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:06.597753+0000","last_peered":"2024-04-22T08:37:06.597753+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:06.597753+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":292,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19653","reported_seq":27334,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.573439+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:06.573439+0000","last_peered":"2024-04-22T08:37:06.573439+0000","last_clean":"2024-04-22T08:37:06.573439+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:06.573439+0000","last_undegraded":"2024-04-22T08:37:06.573439+0000","last_fullsized":"2024-04-22T08:37:06.573439+0000","mapping_epoch":38,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152210,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19651,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19859","reported_seq":20617,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.586153+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:07.586153+0000","last_peered":"2024-04-22T08:37:07.586153+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:07.586153+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1159785,"num_objects":284,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19858,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19478","reported_seq":20154,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.507671+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:07.507671+0000","last_peered":"2024-04-22T08:37:07.507671+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:07.507671+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19764","reported_seq":20369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.596562+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:07.596562+0000","last_peered":"2024-04-22T08:37:07.596562+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:07.596562+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1123538,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19762,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19755","reported_seq":20472,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.543447+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:07.543447+0000","last_peered":"2024-04-22T08:37:07.543447+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:07.543447+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014180,"num_objects":251,"num_object_clones":0,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19752,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17575789,"num_objects":4303,"num_object_clones":0,"num_object_copies":8606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289401,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16880,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221319168,"data_stored":221251881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117117,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495816,"kb_used_data":131136,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93876024,"statfs":{"total":96636764160,"available":96129048576,"internally_reserved":0,"allocated":134283264,"data_stored":134046842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373419727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.23999999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.315}]}]},{"osd":1,"up_from":33,"seq":141733920879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500520,"kb_used_data":13592,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93871320,"statfs":{"total":96636764160,"available":96124231680,"internally_reserved":0,"allocated":13918208,"data_stored":13633080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313346,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529536,"kb_used_data":74668,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93842304,"statfs":{"total":96636764160,"available":96094519296,"internally_reserved":0,"allocated":76460032,"data_stored":76234144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.1259999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":6.1100000000000003},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74874880,"data_stored":74871401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12849152,"data_stored":12792895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133595136,"data_stored":133587585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:10.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:11.148 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:11.150 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:11.195 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-04-22T08:37:10.412385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17575789,"num_objects":4303,"num_object_clones":0,"num_object_copies":8606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289401,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16880,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1525872,"kb_used_data":219396,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281589648,"statfs":{"total":289910292480,"available":288347799552,"internally_reserved":0,"allocated":224661504,"data_stored":223914066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1337773166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19361792,"num_objects":-4727,"num_object_clones":0,"num_object_copies":-9454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002881"},"pg_stats":[{"pgid":"2.f","version":"41'19914","reported_seq":20636,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.476616+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:07.476616+0000","last_peered":"2024-04-22T08:37:07.476616+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:07.476616+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1040384,"num_objects":254,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19914,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3214}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'19931","reported_seq":20053,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.456465+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:06.456465+0000","last_peered":"2024-04-22T08:37:06.456465+0000","last_clean":"2024-04-22T08:37:06.456465+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:06.456465+0000","last_undegraded":"2024-04-22T08:37:06.456465+0000","last_fullsized":"2024-04-22T08:37:06.456465+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232896,"num_objects":301,"num_object_clones":0,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16987,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19586","reported_seq":20123,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.499564+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:06.499564+0000","last_peered":"2024-04-22T08:37:06.499564+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:06.499564+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1134616,"num_objects":278,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16413,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19833","reported_seq":19870,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.589231+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:06.589231+0000","last_peered":"2024-04-22T08:37:06.589231+0000","last_clean":"2024-04-22T08:37:06.589231+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:06.589231+0000","last_undegraded":"2024-04-22T08:37:06.589231+0000","last_fullsized":"2024-04-22T08:37:06.589231+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1150976,"num_objects":281,"num_object_clones":0,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19833,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19686","reported_seq":19828,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.496477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:07.496477+0000","last_peered":"2024-04-22T08:37:07.496477+0000","last_clean":"2024-04-22T08:37:07.496477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:07.496477+0000","last_undegraded":"2024-04-22T08:37:07.496477+0000","last_fullsized":"2024-04-22T08:37:07.496477+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":954368,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16778,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19795","reported_seq":27736,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.448202+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:07.448202+0000","last_peered":"2024-04-22T08:37:07.448202+0000","last_clean":"2024-04-22T08:37:07.448202+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:07.448202+0000","last_undegraded":"2024-04-22T08:37:07.448202+0000","last_fullsized":"2024-04-22T08:37:07.448202+0000","mapping_epoch":40,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16664,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19623","reported_seq":27366,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.274855+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:07.274855+0000","last_peered":"2024-04-22T08:37:07.274855+0000","last_clean":"2024-04-22T08:37:07.274855+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:07.274855+0000","last_undegraded":"2024-04-22T08:37:07.274855+0000","last_fullsized":"2024-04-22T08:37:07.274855+0000","mapping_epoch":36,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":987136,"num_objects":241,"num_object_clones":0,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15991,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19546","reported_seq":19697,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.524244+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:07.524244+0000","last_peered":"2024-04-22T08:37:07.524244+0000","last_clean":"2024-04-22T08:37:07.524244+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:07.524244+0000","last_undegraded":"2024-04-22T08:37:07.524244+0000","last_fullsized":"2024-04-22T08:37:07.524244+0000","mapping_epoch":27,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1064960,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16194,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'19953","reported_seq":20413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.482380+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:06.482380+0000","last_peered":"2024-04-22T08:37:06.482380+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:06.482380+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1098962,"num_objects":270,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19893,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19913","reported_seq":20797,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.577367+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:07.577367+0000","last_peered":"2024-04-22T08:37:07.577367+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:07.577367+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119442,"num_objects":275,"num_object_clones":0,"num_object_copies":550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19911,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19637","reported_seq":26131,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:06.597753+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:06.597753+0000","last_peered":"2024-04-22T08:37:06.597753+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:06.597753+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":292,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-290}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19653","reported_seq":27334,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:06.573439+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:06.573439+0000","last_peered":"2024-04-22T08:37:06.573439+0000","last_clean":"2024-04-22T08:37:06.573439+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:06.573439+0000","last_undegraded":"2024-04-22T08:37:06.573439+0000","last_fullsized":"2024-04-22T08:37:06.573439+0000","mapping_epoch":38,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152210,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19651,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19859","reported_seq":20617,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.586153+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:07.586153+0000","last_peered":"2024-04-22T08:37:07.586153+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:07.586153+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1159785,"num_objects":284,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19858,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19478","reported_seq":20154,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.507671+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:07.507671+0000","last_peered":"2024-04-22T08:37:07.507671+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:07.507671+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19764","reported_seq":20369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.596562+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:07.596562+0000","last_peered":"2024-04-22T08:37:07.596562+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:07.596562+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1123538,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19762,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19755","reported_seq":20472,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.543447+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:07.543447+0000","last_peered":"2024-04-22T08:37:07.543447+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:07.543447+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014180,"num_objects":251,"num_object_clones":0,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19752,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17575789,"num_objects":4303,"num_object_clones":0,"num_object_copies":8606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289401,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16880,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221319168,"data_stored":221251881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117117,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495816,"kb_used_data":131136,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93876024,"statfs":{"total":96636764160,"available":96129048576,"internally_reserved":0,"allocated":134283264,"data_stored":134046842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373419727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:36:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.5169999999999999,"15min":1.1639999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":15.6,"5min":15.6,"15min":15.6},"last":0.23999999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.603,"15min":0.96699999999999997},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":5.9450000000000003,"5min":18.32,"15min":18.32},"last":0.315}]}]},{"osd":1,"up_from":33,"seq":141733920879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500520,"kb_used_data":13592,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93871320,"statfs":{"total":96636764160,"available":96124231680,"internally_reserved":0,"allocated":13918208,"data_stored":13633080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313346,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529536,"kb_used_data":74668,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93842304,"statfs":{"total":96636764160,"available":96094519296,"internally_reserved":0,"allocated":76460032,"data_stored":76234144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.1259999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":6.1100000000000003},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74874880,"data_stored":74871401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12849152,"data_stored":12792895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133595136,"data_stored":133587585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:11.196 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:11.196 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'19914", 'reported_seq': 20636, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.476616+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:07.476616+0000', 'last_peered': '2024-04-22T08:37:07.476616+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:07.476616+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1040384, 'num_objects': 254, 'num_object_clones': 0, 'num_object_copies': 508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19914, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3214}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19586", 'reported_seq': 20123, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:06.499564+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:06.499564+0000', 'last_peered': '2024-04-22T08:37:06.499564+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:06.499564+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9500", 'ondisk_log_start': "37'9500", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1134616, 'num_objects': 278, 'num_object_clones': 0, 'num_object_copies': 556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 278, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16413, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'19953", 'reported_seq': 20413, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:06.482380+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:06.482380+0000', 'last_peered': '2024-04-22T08:37:06.482380+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:06.482380+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1098962, 'num_objects': 270, 'num_object_clones': 0, 'num_object_copies': 540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19893, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'19913", 'reported_seq': 20797, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.577367+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:07.577367+0000', 'last_peered': '2024-04-22T08:37:07.577367+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:07.577367+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1119442, 'num_objects': 275, 'num_object_clones': 0, 'num_object_copies': 550, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19911, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3096}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19637", 'reported_seq': 26131, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:06.597753+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:37:06.597753+0000', 'last_peered': '2024-04-22T08:37:06.597753+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:37:06.597753+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9600", 'ondisk_log_start': "37'9600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1196032, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16091, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -290}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:11.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'19859", 'reported_seq': 20617, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.586153+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:07.586153+0000', 'last_peered': '2024-04-22T08:37:07.586153+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:07.586153+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1159785, 'num_objects': 284, 'num_object_clones': 0, 'num_object_copies': 568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19858, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3101}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19478", 'reported_seq': 20154, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.507671+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:07.507671+0000', 'last_peered': '2024-04-22T08:37:07.507671+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:07.507671+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9400", 'ondisk_log_start': "37'9400", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1114112, 'num_objects': 272, 'num_object_clones': 0, 'num_object_copies': 544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15709, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'19764", 'reported_seq': 20369, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.596562+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:37:07.596562+0000', 'last_peered': '2024-04-22T08:37:07.596562+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:07.596562+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1123538, 'num_objects': 276, 'num_object_clones': 0, 'num_object_copies': 552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19762, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:11.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'19755", 'reported_seq': 20472, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:07.543447+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:07.543447+0000', 'last_peered': '2024-04-22T08:37:07.543447+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:07.543447+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1014180, 'num_objects': 251, 'num_object_clones': 0, 'num_object_copies': 502, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19752, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:11.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:12.044 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28727/8606 objects degraded (333.802%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":666,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":4303,"data_bytes":17575789,"bytes_used":1562492928,"bytes_avail":288347799552,"bytes_total":289910292480,"degraded_objects":28727,"degraded_total":8606,"degraded_ratio":3.3380199860562398,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":273,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":393},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:12.097 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:12.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:12.946 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:13.079 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-22T08:37:12.412821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":14008173,"num_objects":3432,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290272,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160997,"ondisk_log_size":160997,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1508976,"kb_used_data":214784,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281606544,"statfs":{"total":289910292480,"available":288365101056,"internally_reserved":0,"allocated":219938816,"data_stored":219191378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,1,11,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22929408,"num_objects":-5598,"num_object_clones":0,"num_object_copies":-11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"2.f","version":"41'19914","reported_seq":20636,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.476616+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:07.476616+0000","last_peered":"2024-04-22T08:37:07.476616+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:07.476616+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1040384,"num_objects":254,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19914,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3214}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20090","reported_seq":20212,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.507708+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:11.507708+0000","last_peered":"2024-04-22T08:37:11.507708+0000","last_clean":"2024-04-22T08:37:11.507708+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:11.507708+0000","last_undegraded":"2024-04-22T08:37:11.507708+0000","last_fullsized":"2024-04-22T08:37:11.507708+0000","mapping_epoch":27,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581632,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19723","reported_seq":20258,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.583919+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:11.583919+0000","last_peered":"2024-04-22T08:37:11.583919+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:11.583919+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573464,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16550,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19975","reported_seq":20012,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.539878+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:11.539878+0000","last_peered":"2024-04-22T08:37:11.539878+0000","last_clean":"2024-04-22T08:37:11.539878+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:11.539878+0000","last_undegraded":"2024-04-22T08:37:11.539878+0000","last_fullsized":"2024-04-22T08:37:11.539878+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569344,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19686","reported_seq":19828,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.496477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:07.496477+0000","last_peered":"2024-04-22T08:37:07.496477+0000","last_clean":"2024-04-22T08:37:07.496477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:07.496477+0000","last_undegraded":"2024-04-22T08:37:07.496477+0000","last_fullsized":"2024-04-22T08:37:07.496477+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":954368,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16778,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19795","reported_seq":27736,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.448202+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:07.448202+0000","last_peered":"2024-04-22T08:37:07.448202+0000","last_clean":"2024-04-22T08:37:07.448202+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:07.448202+0000","last_undegraded":"2024-04-22T08:37:07.448202+0000","last_fullsized":"2024-04-22T08:37:07.448202+0000","mapping_epoch":40,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16664,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19623","reported_seq":27366,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.274855+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:07.274855+0000","last_peered":"2024-04-22T08:37:07.274855+0000","last_clean":"2024-04-22T08:37:07.274855+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:07.274855+0000","last_undegraded":"2024-04-22T08:37:07.274855+0000","last_fullsized":"2024-04-22T08:37:07.274855+0000","mapping_epoch":36,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":987136,"num_objects":241,"num_object_clones":0,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15991,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19546","reported_seq":19697,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:07.524244+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:07.524244+0000","last_peered":"2024-04-22T08:37:07.524244+0000","last_clean":"2024-04-22T08:37:07.524244+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:07.524244+0000","last_undegraded":"2024-04-22T08:37:07.524244+0000","last_fullsized":"2024-04-22T08:37:07.524244+0000","mapping_epoch":27,"log_start":"37'9500","ondisk_log_start":"37'9500","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1064960,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16194,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20088","reported_seq":20548,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.475988+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:11.475988+0000","last_peered":"2024-04-22T08:37:11.475988+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:11.475988+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546002,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20028,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'19913","reported_seq":20797,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.577367+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:07.577367+0000","last_peered":"2024-04-22T08:37:07.577367+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:07.577367+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119442,"num_objects":275,"num_object_clones":0,"num_object_copies":550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19911,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19796","reported_seq":26606,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.631397+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:11.631397+0000","last_peered":"2024-04-22T08:37:11.631397+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:11.631397+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544768,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19792","reported_seq":27474,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.625645+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:11.625645+0000","last_peered":"2024-04-22T08:37:11.625645+0000","last_clean":"2024-04-22T08:37:11.625645+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:11.625645+0000","last_undegraded":"2024-04-22T08:37:11.625645+0000","last_fullsized":"2024-04-22T08:37:11.625645+0000","mapping_epoch":38,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582866,"num_objects":144,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19790,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'19859","reported_seq":20617,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.586153+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:07.586153+0000","last_peered":"2024-04-22T08:37:07.586153+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:07.586153+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1159785,"num_objects":284,"num_object_clones":0,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19858,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19478","reported_seq":20154,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.507671+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:07.507671+0000","last_peered":"2024-04-22T08:37:07.507671+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:07.507671+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9400","ondisk_log_start":"37'9400","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19764","reported_seq":20369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.596562+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:07.596562+0000","last_peered":"2024-04-22T08:37:07.596562+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:07.596562+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1123538,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19762,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19755","reported_seq":20472,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:07.543447+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:07.543447+0000","last_peered":"2024-04-22T08:37:07.543447+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:07.543447+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1014180,"num_objects":251,"num_object_clones":0,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19752,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":14008173,"num_objects":3432,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290272,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216588288,"data_stored":216521001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160997,"ondisk_log_size":160997,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117118,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478920,"kb_used_data":126524,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93892920,"statfs":{"total":96636764160,"available":96146350080,"internally_reserved":0,"allocated":129560576,"data_stored":129324154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":6.3860000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":6.431}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":6.6239999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500520,"kb_used_data":13592,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93871320,"statfs":{"total":96636764160,"available":96124231680,"internally_reserved":0,"allocated":13918208,"data_stored":13633080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313346,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529536,"kb_used_data":74668,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93842304,"statfs":{"total":96636764160,"available":96094519296,"internally_reserved":0,"allocated":76460032,"data_stored":76234144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.4590000000000001,"15min":0.87},"min":{"1min":0.223,"5min":0.16,"15min":0.16},"max":{"1min":10.972,"5min":11.368,"15min":11.368},"last":3.1259999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.84999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2489999999999997,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":6.1100000000000003},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74874880,"data_stored":74871401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12849152,"data_stored":12792895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128864256,"data_stored":128856705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:16.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:16.435 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Removed 160037 objects 2024-04-22T08:37:16.435 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi049.stdout:Clean up completed and total clean up time :363.513 2024-04-22T08:37:16.576 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:16.576 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:16.607 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-22T08:37:16.414706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1478520,"kb_used_data":200724,"kb_used_omap":12,"kb_used_meta":1277747,"kb_avail":281637000,"statfs":{"total":289910292480,"available":288396288000,"internally_reserved":0,"allocated":205541376,"data_stored":204793938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1308413038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,3,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19140608,"num_objects":-4673,"num_object_clones":0,"num_object_copies":-9346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003738"},"pg_stats":[{"pgid":"2.f","version":"41'20077","reported_seq":20799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.625117+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:12.625117+0000","last_peered":"2024-04-22T08:37:12.625117+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:12.625117+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372736,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20077,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3377}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20090","reported_seq":20212,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.507708+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:11.507708+0000","last_peered":"2024-04-22T08:37:11.507708+0000","last_clean":"2024-04-22T08:37:11.507708+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:11.507708+0000","last_undegraded":"2024-04-22T08:37:11.507708+0000","last_fullsized":"2024-04-22T08:37:11.507708+0000","mapping_epoch":27,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581632,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19723","reported_seq":20258,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.583919+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:11.583919+0000","last_peered":"2024-04-22T08:37:11.583919+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:11.583919+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573464,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16550,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19975","reported_seq":20012,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.539878+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:11.539878+0000","last_peered":"2024-04-22T08:37:11.539878+0000","last_clean":"2024-04-22T08:37:11.539878+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:11.539878+0000","last_undegraded":"2024-04-22T08:37:11.539878+0000","last_fullsized":"2024-04-22T08:37:11.539878+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569344,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19818","reported_seq":19963,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.556021+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:12.556021+0000","last_peered":"2024-04-22T08:37:12.556021+0000","last_clean":"2024-04-22T08:37:12.556021+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:12.556021+0000","last_undegraded":"2024-04-22T08:37:12.556021+0000","last_fullsized":"2024-04-22T08:37:12.556021+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413696,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16910,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19940","reported_seq":27882,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.485441+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:12.485441+0000","last_peered":"2024-04-22T08:37:12.485441+0000","last_clean":"2024-04-22T08:37:12.485441+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:12.485441+0000","last_undegraded":"2024-04-22T08:37:12.485441+0000","last_fullsized":"2024-04-22T08:37:12.485441+0000","mapping_epoch":40,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438272,"num_objects":107,"num_object_clones":0,"num_object_copies":214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19755","reported_seq":27498,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.470432+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:12.470432+0000","last_peered":"2024-04-22T08:37:12.470432+0000","last_clean":"2024-04-22T08:37:12.470432+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:12.470432+0000","last_undegraded":"2024-04-22T08:37:12.470432+0000","last_fullsized":"2024-04-22T08:37:12.470432+0000","mapping_epoch":36,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446464,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16123,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19692","reported_seq":19843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.638133+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:12.638133+0000","last_peered":"2024-04-22T08:37:12.638133+0000","last_clean":"2024-04-22T08:37:12.638133+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:12.638133+0000","last_undegraded":"2024-04-22T08:37:12.638133+0000","last_fullsized":"2024-04-22T08:37:12.638133+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":114,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16340,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20088","reported_seq":20548,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.475988+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:11.475988+0000","last_peered":"2024-04-22T08:37:11.475988+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:11.475988+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546002,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20028,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20055","reported_seq":20939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.691829+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:12.691829+0000","last_peered":"2024-04-22T08:37:12.691829+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:12.691829+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537810,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20053,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3238}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19796","reported_seq":26606,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.631397+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:11.631397+0000","last_peered":"2024-04-22T08:37:11.631397+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:11.631397+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544768,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19792","reported_seq":27474,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.625645+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:11.625645+0000","last_peered":"2024-04-22T08:37:11.625645+0000","last_clean":"2024-04-22T08:37:11.625645+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:11.625645+0000","last_undegraded":"2024-04-22T08:37:11.625645+0000","last_fullsized":"2024-04-22T08:37:11.625645+0000","mapping_epoch":38,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582866,"num_objects":144,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19790,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20034","reported_seq":20789,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.686787+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:12.686787+0000","last_peered":"2024-04-22T08:37:12.686787+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:12.686787+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442985,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19637","reported_seq":20313,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.680110+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:12.680110+0000","last_peered":"2024-04-22T08:37:12.680110+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:12.680110+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462848,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15868,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19925","reported_seq":20529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.681307+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:12.681307+0000","last_peered":"2024-04-22T08:37:12.681307+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:12.681307+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464082,"num_objects":115,"num_object_clones":0,"num_object_copies":230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19923,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19898","reported_seq":20615,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.684406+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:12.684406+0000","last_peered":"2024-04-22T08:37:12.684406+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:12.684406+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428452,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19895,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202100736,"data_stored":202033449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117118,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478920,"kb_used_data":126524,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93892920,"statfs":{"total":96636764160,"available":96146350080,"internally_reserved":0,"allocated":129560576,"data_stored":129324154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":6.3860000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":6.431}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":6.6239999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485400,"kb_used_data":6672,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93886440,"statfs":{"total":96636764160,"available":96139714560,"internally_reserved":0,"allocated":6832128,"data_stored":6547000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45900000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313347,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514200,"kb_used_data":67528,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93857640,"statfs":{"total":96636764160,"available":96110223360,"internally_reserved":0,"allocated":69148672,"data_stored":68922784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":457371347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":1.764},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.401},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67563520,"data_stored":67560041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5672960,"data_stored":5616703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128864256,"data_stored":128856705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:16.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:16.891 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:16.911 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-22T08:37:16.414706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1478520,"kb_used_data":200724,"kb_used_omap":12,"kb_used_meta":1277747,"kb_avail":281637000,"statfs":{"total":289910292480,"available":288396288000,"internally_reserved":0,"allocated":205541376,"data_stored":204793938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1308413038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,3,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19140608,"num_objects":-4673,"num_object_clones":0,"num_object_copies":-9346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003738"},"pg_stats":[{"pgid":"2.f","version":"41'20077","reported_seq":20799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.625117+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:12.625117+0000","last_peered":"2024-04-22T08:37:12.625117+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:12.625117+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372736,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20077,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3377}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20090","reported_seq":20212,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.507708+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:11.507708+0000","last_peered":"2024-04-22T08:37:11.507708+0000","last_clean":"2024-04-22T08:37:11.507708+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:11.507708+0000","last_undegraded":"2024-04-22T08:37:11.507708+0000","last_fullsized":"2024-04-22T08:37:11.507708+0000","mapping_epoch":27,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581632,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19723","reported_seq":20258,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.583919+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:11.583919+0000","last_peered":"2024-04-22T08:37:11.583919+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:11.583919+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573464,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16550,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19975","reported_seq":20012,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.539878+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:11.539878+0000","last_peered":"2024-04-22T08:37:11.539878+0000","last_clean":"2024-04-22T08:37:11.539878+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:11.539878+0000","last_undegraded":"2024-04-22T08:37:11.539878+0000","last_fullsized":"2024-04-22T08:37:11.539878+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569344,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19818","reported_seq":19963,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.556021+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:12.556021+0000","last_peered":"2024-04-22T08:37:12.556021+0000","last_clean":"2024-04-22T08:37:12.556021+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:12.556021+0000","last_undegraded":"2024-04-22T08:37:12.556021+0000","last_fullsized":"2024-04-22T08:37:12.556021+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413696,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16910,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19940","reported_seq":27882,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.485441+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:12.485441+0000","last_peered":"2024-04-22T08:37:12.485441+0000","last_clean":"2024-04-22T08:37:12.485441+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:12.485441+0000","last_undegraded":"2024-04-22T08:37:12.485441+0000","last_fullsized":"2024-04-22T08:37:12.485441+0000","mapping_epoch":40,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438272,"num_objects":107,"num_object_clones":0,"num_object_copies":214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19755","reported_seq":27498,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.470432+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:12.470432+0000","last_peered":"2024-04-22T08:37:12.470432+0000","last_clean":"2024-04-22T08:37:12.470432+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:12.470432+0000","last_undegraded":"2024-04-22T08:37:12.470432+0000","last_fullsized":"2024-04-22T08:37:12.470432+0000","mapping_epoch":36,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446464,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16123,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19692","reported_seq":19843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.638133+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:12.638133+0000","last_peered":"2024-04-22T08:37:12.638133+0000","last_clean":"2024-04-22T08:37:12.638133+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:12.638133+0000","last_undegraded":"2024-04-22T08:37:12.638133+0000","last_fullsized":"2024-04-22T08:37:12.638133+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":114,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16340,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20088","reported_seq":20548,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.475988+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:11.475988+0000","last_peered":"2024-04-22T08:37:11.475988+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:11.475988+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546002,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20028,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20055","reported_seq":20939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.691829+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:12.691829+0000","last_peered":"2024-04-22T08:37:12.691829+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:12.691829+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537810,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20053,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3238}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19796","reported_seq":26606,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.631397+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:11.631397+0000","last_peered":"2024-04-22T08:37:11.631397+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:11.631397+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544768,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19792","reported_seq":27474,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.625645+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:11.625645+0000","last_peered":"2024-04-22T08:37:11.625645+0000","last_clean":"2024-04-22T08:37:11.625645+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:11.625645+0000","last_undegraded":"2024-04-22T08:37:11.625645+0000","last_fullsized":"2024-04-22T08:37:11.625645+0000","mapping_epoch":38,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582866,"num_objects":144,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19790,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20034","reported_seq":20789,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.686787+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:12.686787+0000","last_peered":"2024-04-22T08:37:12.686787+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:12.686787+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442985,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19637","reported_seq":20313,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.680110+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:12.680110+0000","last_peered":"2024-04-22T08:37:12.680110+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:12.680110+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462848,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15868,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19925","reported_seq":20529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.681307+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:12.681307+0000","last_peered":"2024-04-22T08:37:12.681307+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:12.681307+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464082,"num_objects":115,"num_object_clones":0,"num_object_copies":230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19923,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19898","reported_seq":20615,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.684406+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:12.684406+0000","last_peered":"2024-04-22T08:37:12.684406+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:12.684406+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428452,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19895,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202100736,"data_stored":202033449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117118,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478920,"kb_used_data":126524,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93892920,"statfs":{"total":96636764160,"available":96146350080,"internally_reserved":0,"allocated":129560576,"data_stored":129324154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":6.3860000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":6.431}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":6.6239999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485400,"kb_used_data":6672,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93886440,"statfs":{"total":96636764160,"available":96139714560,"internally_reserved":0,"allocated":6832128,"data_stored":6547000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45900000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313347,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514200,"kb_used_data":67528,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93857640,"statfs":{"total":96636764160,"available":96110223360,"internally_reserved":0,"allocated":69148672,"data_stored":68922784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":457371347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":1.764},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.401},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67563520,"data_stored":67560041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5672960,"data_stored":5616703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128864256,"data_stored":128856705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:16.912 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:16.912 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20077", 'reported_seq': 20799, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.625117+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:12.625117+0000', 'last_peered': '2024-04-22T08:37:12.625117+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:12.625117+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 372736, 'num_objects': 91, 'num_object_clones': 0, 'num_object_copies': 182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 91, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20077, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3377}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.912 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19723", 'reported_seq': 20258, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:11.583919+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:11.583919+0000', 'last_peered': '2024-04-22T08:37:11.583919+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:11.583919+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 573464, 'num_objects': 141, 'num_object_clones': 0, 'num_object_copies': 282, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 141, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16550, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3411}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20088", 'reported_seq': 20548, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:11.475988+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:11.475988+0000', 'last_peered': '2024-04-22T08:37:11.475988+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:11.475988+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 546002, 'num_objects': 135, 'num_object_clones': 0, 'num_object_copies': 270, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20028, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20055", 'reported_seq': 20939, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.691829+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:12.691829+0000', 'last_peered': '2024-04-22T08:37:12.691829+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:12.691829+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 537810, 'num_objects': 133, 'num_object_clones': 0, 'num_object_copies': 266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20053, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3238}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19796", 'reported_seq': 26606, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:11.631397+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:37:11.631397+0000', 'last_peered': '2024-04-22T08:37:11.631397+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:37:11.631397+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 544768, 'num_objects': 133, 'num_object_clones': 0, 'num_object_copies': 266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 424, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16250, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3328, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -291}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20034", 'reported_seq': 20789, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.686787+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:12.686787+0000', 'last_peered': '2024-04-22T08:37:12.686787+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:12.686787+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 442985, 'num_objects': 109, 'num_object_clones': 0, 'num_object_copies': 218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3276}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:16.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19637", 'reported_seq': 20313, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.680110+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:12.680110+0000', 'last_peered': '2024-04-22T08:37:12.680110+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:12.680110+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9600", 'ondisk_log_start': "37'9600", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 462848, 'num_objects': 113, 'num_object_clones': 0, 'num_object_copies': 226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15868, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:16.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'19925", 'reported_seq': 20529, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.681307+0000', 'last_change': '2024-04-22T08:27:56.660579+0000', 'last_active': '2024-04-22T08:37:12.681307+0000', 'last_peered': '2024-04-22T08:37:12.681307+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:12.681307+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 464082, 'num_objects': 115, 'num_object_clones': 0, 'num_object_copies': 230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19923, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:16.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'19898", 'reported_seq': 20615, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:12.684406+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:12.684406+0000', 'last_peered': '2024-04-22T08:37:12.684406+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:12.684406+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 428452, 'num_objects': 108, 'num_object_clones': 0, 'num_object_copies': 216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19895, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:16.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:17.252 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:17.253 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28569/3868 objects degraded (738.599%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":672,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":1934,"data_bytes":7872365,"bytes_used":1514004480,"bytes_avail":288396288000,"bytes_total":289910292480,"degraded_objects":28569,"degraded_total":3868,"degraded_ratio":7.3859875904860397,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":312,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":389},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:17.270 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:17.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:17.563 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-22T08:37:16.414706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1478520,"kb_used_data":200724,"kb_used_omap":12,"kb_used_meta":1277747,"kb_avail":281637000,"statfs":{"total":289910292480,"available":288396288000,"internally_reserved":0,"allocated":205541376,"data_stored":204793938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1308413038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,3,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19140608,"num_objects":-4673,"num_object_clones":0,"num_object_copies":-9346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003738"},"pg_stats":[{"pgid":"2.f","version":"41'20077","reported_seq":20799,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.625117+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:12.625117+0000","last_peered":"2024-04-22T08:37:12.625117+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:12.625117+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372736,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20077,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3377}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20090","reported_seq":20212,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.507708+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:11.507708+0000","last_peered":"2024-04-22T08:37:11.507708+0000","last_clean":"2024-04-22T08:37:11.507708+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:11.507708+0000","last_undegraded":"2024-04-22T08:37:11.507708+0000","last_fullsized":"2024-04-22T08:37:11.507708+0000","mapping_epoch":27,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581632,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19723","reported_seq":20258,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.583919+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:11.583919+0000","last_peered":"2024-04-22T08:37:11.583919+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:11.583919+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573464,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16550,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'19975","reported_seq":20012,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.539878+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:11.539878+0000","last_peered":"2024-04-22T08:37:11.539878+0000","last_clean":"2024-04-22T08:37:11.539878+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:11.539878+0000","last_undegraded":"2024-04-22T08:37:11.539878+0000","last_fullsized":"2024-04-22T08:37:11.539878+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569344,"num_objects":139,"num_object_clones":0,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19818","reported_seq":19963,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.556021+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:12.556021+0000","last_peered":"2024-04-22T08:37:12.556021+0000","last_clean":"2024-04-22T08:37:12.556021+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:12.556021+0000","last_undegraded":"2024-04-22T08:37:12.556021+0000","last_fullsized":"2024-04-22T08:37:12.556021+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413696,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16910,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'19940","reported_seq":27882,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.485441+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:12.485441+0000","last_peered":"2024-04-22T08:37:12.485441+0000","last_clean":"2024-04-22T08:37:12.485441+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:12.485441+0000","last_undegraded":"2024-04-22T08:37:12.485441+0000","last_fullsized":"2024-04-22T08:37:12.485441+0000","mapping_epoch":40,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438272,"num_objects":107,"num_object_clones":0,"num_object_copies":214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19755","reported_seq":27498,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.470432+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:12.470432+0000","last_peered":"2024-04-22T08:37:12.470432+0000","last_clean":"2024-04-22T08:37:12.470432+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:12.470432+0000","last_undegraded":"2024-04-22T08:37:12.470432+0000","last_fullsized":"2024-04-22T08:37:12.470432+0000","mapping_epoch":36,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446464,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16123,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19692","reported_seq":19843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:12.638133+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:12.638133+0000","last_peered":"2024-04-22T08:37:12.638133+0000","last_clean":"2024-04-22T08:37:12.638133+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:12.638133+0000","last_undegraded":"2024-04-22T08:37:12.638133+0000","last_fullsized":"2024-04-22T08:37:12.638133+0000","mapping_epoch":27,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":114,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16340,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20088","reported_seq":20548,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.475988+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:11.475988+0000","last_peered":"2024-04-22T08:37:11.475988+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:11.475988+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546002,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20028,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20055","reported_seq":20939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.691829+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:12.691829+0000","last_peered":"2024-04-22T08:37:12.691829+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:12.691829+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537810,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20053,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3238}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.578025+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:35:02.578025+0000","last_peered":"2024-04-22T08:35:02.578025+0000","last_clean":"2024-04-22T08:35:02.578025+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:35:02.578025+0000","last_undegraded":"2024-04-22T08:35:02.578025+0000","last_fullsized":"2024-04-22T08:35:02.578025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19796","reported_seq":26606,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:11.631397+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:11.631397+0000","last_peered":"2024-04-22T08:37:11.631397+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:11.631397+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544768,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19792","reported_seq":27474,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:11.625645+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:11.625645+0000","last_peered":"2024-04-22T08:37:11.625645+0000","last_clean":"2024-04-22T08:37:11.625645+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:11.625645+0000","last_undegraded":"2024-04-22T08:37:11.625645+0000","last_fullsized":"2024-04-22T08:37:11.625645+0000","mapping_epoch":38,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582866,"num_objects":144,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19790,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20034","reported_seq":20789,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.686787+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:12.686787+0000","last_peered":"2024-04-22T08:37:12.686787+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:12.686787+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442985,"num_objects":109,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19637","reported_seq":20313,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.680110+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:12.680110+0000","last_peered":"2024-04-22T08:37:12.680110+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:12.680110+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9600","ondisk_log_start":"37'9600","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462848,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15868,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'19925","reported_seq":20529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.681307+0000","last_change":"2024-04-22T08:27:56.660579+0000","last_active":"2024-04-22T08:37:12.681307+0000","last_peered":"2024-04-22T08:37:12.681307+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:12.681307+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464082,"num_objects":115,"num_object_clones":0,"num_object_copies":230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19923,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'19898","reported_seq":20615,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:12.684406+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:12.684406+0000","last_peered":"2024-04-22T08:37:12.684406+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:12.684406+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428452,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19895,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7872365,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291770,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17038,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202100736,"data_stored":202033449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160995,"ondisk_log_size":160995,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117118,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478920,"kb_used_data":126524,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93892920,"statfs":{"total":96636764160,"available":96146350080,"internally_reserved":0,"allocated":129560576,"data_stored":129324154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":6.3860000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":6.431}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":6.6239999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485400,"kb_used_data":6672,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93886440,"statfs":{"total":96636764160,"available":96139714560,"internally_reserved":0,"allocated":6832128,"data_stored":6547000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97799999999999998,"15min":0.72999999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9960000000000004,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.919,"5min":1.046,"15min":0.97299999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.1589999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.103,"15min":1.3919999999999999},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":4.8929999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1160000000000001,"15min":1.244},"min":{"1min":0.26700000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":4.4489999999999998,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45900000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313347,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514200,"kb_used_data":67528,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93857640,"statfs":{"total":96636764160,"available":96110223360,"internally_reserved":0,"allocated":69148672,"data_stored":68922784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":457371347},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":1.764},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":1.401},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67563520,"data_stored":67560041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5672960,"data_stored":5616703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128864256,"data_stored":128856705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:20.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:21.778 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:21.833 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-04-22T08:37:20.430592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17328,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477648,"kb_used_data":183480,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637872,"statfs":{"total":289910292480,"available":288397180928,"internally_reserved":0,"allocated":187883520,"data_stored":187140154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,11,8,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14000152,"num_objects":-3419,"num_object_clones":0,"num_object_copies":-6838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.017771"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442097+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:16.442097+0000","last_peered":"2024-04-22T08:37:16.442097+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:16.442097+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20354,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.423456+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:16.423456+0000","last_peered":"2024-04-22T08:37:16.423456+0000","last_clean":"2024-04-22T08:37:16.423456+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:16.423456+0000","last_undegraded":"2024-04-22T08:37:16.423456+0000","last_fullsized":"2024-04-22T08:37:16.423456+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20401,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.404980+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:16.404980+0000","last_peered":"2024-04-22T08:37:16.404980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:16.404980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20151,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.435471+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:16.435471+0000","last_peered":"2024-04-22T08:37:16.435471+0000","last_clean":"2024-04-22T08:37:16.435471+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:16.435471+0000","last_undegraded":"2024-04-22T08:37:16.435471+0000","last_fullsized":"2024-04-22T08:37:16.435471+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20065,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.430696+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:16.430696+0000","last_peered":"2024-04-22T08:37:16.430696+0000","last_clean":"2024-04-22T08:37:16.430696+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:16.430696+0000","last_undegraded":"2024-04-22T08:37:16.430696+0000","last_fullsized":"2024-04-22T08:37:16.430696+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.392589+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:16.392589+0000","last_peered":"2024-04-22T08:37:16.392589+0000","last_clean":"2024-04-22T08:37:16.392589+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:16.392589+0000","last_undegraded":"2024-04-22T08:37:16.392589+0000","last_fullsized":"2024-04-22T08:37:16.392589+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27607,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.428554+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:16.428554+0000","last_peered":"2024-04-22T08:37:16.428554+0000","last_clean":"2024-04-22T08:37:16.428554+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:16.428554+0000","last_undegraded":"2024-04-22T08:37:16.428554+0000","last_fullsized":"2024-04-22T08:37:16.428554+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19958,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.389395+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:16.389395+0000","last_peered":"2024-04-22T08:37:16.389395+0000","last_clean":"2024-04-22T08:37:16.389395+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:16.389395+0000","last_undegraded":"2024-04-22T08:37:16.389395+0000","last_fullsized":"2024-04-22T08:37:16.389395+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20681,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.296081+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:16.296081+0000","last_peered":"2024-04-22T08:37:16.296081+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:16.296081+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21072,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589598+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.589598+0000","last_peered":"2024-04-22T08:37:17.589598+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.589598+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":76,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589533+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.589533+0000","last_peered":"2024-04-22T08:37:17.589533+0000","last_clean":"2024-04-22T08:37:17.589533+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.589533+0000","last_undegraded":"2024-04-22T08:37:17.589533+0000","last_fullsized":"2024-04-22T08:37:17.589533+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27227,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.641493+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:16.641493+0000","last_peered":"2024-04-22T08:37:16.641493+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:16.641493+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3572,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27616,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.396977+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:16.396977+0000","last_peered":"2024-04-22T08:37:16.396977+0000","last_clean":"2024-04-22T08:37:16.396977+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:16.396977+0000","last_undegraded":"2024-04-22T08:37:16.396977+0000","last_fullsized":"2024-04-22T08:37:16.396977+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20901,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442495+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:16.442495+0000","last_peered":"2024-04-22T08:37:16.442495+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:16.442495+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20427,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.411700+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:16.411700+0000","last_peered":"2024-04-22T08:37:16.411700+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:16.411700+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20737,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.716912+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:17.716912+0000","last_peered":"2024-04-22T08:37:17.716912+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:17.716912+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20720,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.434505+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:16.434505+0000","last_peered":"2024-04-22T08:37:16.434505+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:16.434505+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17328,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117119,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478456,"kb_used_data":121968,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893384,"statfs":{"total":96636764160,"available":96146825216,"internally_reserved":0,"allocated":124895232,"data_stored":124658810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.309},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.246}]}]},{"osd":1,"up_from":33,"seq":141733920881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487316,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884524,"statfs":{"total":96636764160,"available":96137752576,"internally_reserved":0,"allocated":405504,"data_stored":124448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313348,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511876,"kb_used_data":61116,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859964,"statfs":{"total":96636764160,"available":96112603136,"internally_reserved":0,"allocated":62582784,"data_stored":62356896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:21.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:22.149 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:22.149 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:22.167 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-04-22T08:37:20.430592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17328,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477648,"kb_used_data":183480,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637872,"statfs":{"total":289910292480,"available":288397180928,"internally_reserved":0,"allocated":187883520,"data_stored":187140154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,11,8,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14000152,"num_objects":-3419,"num_object_clones":0,"num_object_copies":-6838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.017771"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442097+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:16.442097+0000","last_peered":"2024-04-22T08:37:16.442097+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:16.442097+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20354,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.423456+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:16.423456+0000","last_peered":"2024-04-22T08:37:16.423456+0000","last_clean":"2024-04-22T08:37:16.423456+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:16.423456+0000","last_undegraded":"2024-04-22T08:37:16.423456+0000","last_fullsized":"2024-04-22T08:37:16.423456+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20401,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.404980+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:16.404980+0000","last_peered":"2024-04-22T08:37:16.404980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:16.404980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20151,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.435471+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:16.435471+0000","last_peered":"2024-04-22T08:37:16.435471+0000","last_clean":"2024-04-22T08:37:16.435471+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:16.435471+0000","last_undegraded":"2024-04-22T08:37:16.435471+0000","last_fullsized":"2024-04-22T08:37:16.435471+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20065,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.430696+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:16.430696+0000","last_peered":"2024-04-22T08:37:16.430696+0000","last_clean":"2024-04-22T08:37:16.430696+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:16.430696+0000","last_undegraded":"2024-04-22T08:37:16.430696+0000","last_fullsized":"2024-04-22T08:37:16.430696+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.392589+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:16.392589+0000","last_peered":"2024-04-22T08:37:16.392589+0000","last_clean":"2024-04-22T08:37:16.392589+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:16.392589+0000","last_undegraded":"2024-04-22T08:37:16.392589+0000","last_fullsized":"2024-04-22T08:37:16.392589+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27607,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.428554+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:16.428554+0000","last_peered":"2024-04-22T08:37:16.428554+0000","last_clean":"2024-04-22T08:37:16.428554+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:16.428554+0000","last_undegraded":"2024-04-22T08:37:16.428554+0000","last_fullsized":"2024-04-22T08:37:16.428554+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19958,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.389395+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:16.389395+0000","last_peered":"2024-04-22T08:37:16.389395+0000","last_clean":"2024-04-22T08:37:16.389395+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:16.389395+0000","last_undegraded":"2024-04-22T08:37:16.389395+0000","last_fullsized":"2024-04-22T08:37:16.389395+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20681,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.296081+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:16.296081+0000","last_peered":"2024-04-22T08:37:16.296081+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:16.296081+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.581427+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:35:02.581427+0000","last_peered":"2024-04-22T08:35:02.581427+0000","last_clean":"2024-04-22T08:35:02.581427+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:35:02.581427+0000","last_undegraded":"2024-04-22T08:35:02.581427+0000","last_fullsized":"2024-04-22T08:35:02.581427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21072,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589598+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.589598+0000","last_peered":"2024-04-22T08:37:17.589598+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.589598+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":76,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589533+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.589533+0000","last_peered":"2024-04-22T08:37:17.589533+0000","last_clean":"2024-04-22T08:37:17.589533+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.589533+0000","last_undegraded":"2024-04-22T08:37:17.589533+0000","last_fullsized":"2024-04-22T08:37:17.589533+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27227,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.641493+0000","last_change":"2024-04-22T08:35:00.826192+0000","last_active":"2024-04-22T08:37:16.641493+0000","last_peered":"2024-04-22T08:37:16.641493+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.614714+0000","last_became_peered":"2024-04-22T08:27:56.614714+0000","last_unstale":"2024-04-22T08:37:16.641493+0000","last_undegraded":"2024-04-22T08:27:56.177829+0000","last_fullsized":"2024-04-22T08:27:56.089181+0000","mapping_epoch":34,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3572,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27616,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.396977+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:16.396977+0000","last_peered":"2024-04-22T08:37:16.396977+0000","last_clean":"2024-04-22T08:37:16.396977+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:16.396977+0000","last_undegraded":"2024-04-22T08:37:16.396977+0000","last_fullsized":"2024-04-22T08:37:16.396977+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20901,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442495+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:16.442495+0000","last_peered":"2024-04-22T08:37:16.442495+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:16.442495+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20427,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.411700+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:16.411700+0000","last_peered":"2024-04-22T08:37:16.411700+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:16.411700+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.579599+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:35:02.579599+0000","last_peered":"2024-04-22T08:35:02.579599+0000","last_clean":"2024-04-22T08:35:02.579599+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:35:02.579599+0000","last_undegraded":"2024-04-22T08:35:02.579599+0000","last_fullsized":"2024-04-22T08:35:02.579599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20737,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.716912+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:17.716912+0000","last_peered":"2024-04-22T08:37:17.716912+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:17.716912+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20720,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.434505+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:16.434505+0000","last_peered":"2024-04-22T08:37:16.434505+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:16.434505+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17328,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117119,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478456,"kb_used_data":121968,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893384,"statfs":{"total":96636764160,"available":96146825216,"internally_reserved":0,"allocated":124895232,"data_stored":124658810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1040000000000001,"15min":0.997},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":4.9119999999999999,"5min":6.673,"15min":13.635},"last":0.309},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":4.9420000000000002,"5min":6.7670000000000003,"15min":13.721},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.246}]}]},{"osd":1,"up_from":33,"seq":141733920881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487316,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884524,"statfs":{"total":96636764160,"available":96137752576,"internally_reserved":0,"allocated":405504,"data_stored":124448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313348,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511876,"kb_used_data":61116,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859964,"statfs":{"total":96636764160,"available":96112603136,"internally_reserved":0,"allocated":62582784,"data_stored":62356896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20891, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.442097+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:16.442097+0000', 'last_peered': '2024-04-22T08:37:16.442097+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:16.442097+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20401, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.404980+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:16.404980+0000', 'last_peered': '2024-04-22T08:37:16.404980+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:16.404980+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20681, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.296081+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:16.296081+0000', 'last_peered': '2024-04-22T08:37:16.296081+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:16.296081+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:22.169 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21072, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589598+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.589598+0000', 'last_peered': '2024-04-22T08:37:17.589598+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.589598+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'19929", 'reported_seq': 27227, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.641493+0000', 'last_change': '2024-04-22T08:35:00.826192+0000', 'last_active': '2024-04-22T08:37:16.641493+0000', 'last_peered': '2024-04-22T08:37:16.641493+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.614714+0000', 'last_became_peered': '2024-04-22T08:27:56.614714+0000', 'last_unstale': '2024-04-22T08:37:16.641493+0000', 'last_undegraded': '2024-04-22T08:27:56.177829+0000', 'last_fullsized': '2024-04-22T08:27:56.089181+0000', 'mapping_epoch': 34, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3623", 'last_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:52.002961+0000', 'objects_scrubbed': 3541, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T17:46:11.295707+0000', 'scrub_duration': 2.11846286, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 180, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16383, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3572, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -180}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20901, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.442495+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:16.442495+0000', 'last_peered': '2024-04-22T08:37:16.442495+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:16.442495+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20427, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.411700+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:16.411700+0000', 'last_peered': '2024-04-22T08:37:16.411700+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:16.411700+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 20737, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.716912+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:17.716912+0000', 'last_peered': '2024-04-22T08:37:17.716912+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:17.716912+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 309, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:22.170 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20720, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:16.434505+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:16.434505+0000', 'last_peered': '2024-04-22T08:37:16.434505+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:16.434505+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:22.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:22.554 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:22.555 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28279/26 objects degraded (108765.385%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":677,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1513111552,"bytes_avail":288397180928,"bytes_total":289910292480,"degraded_objects":28279,"degraded_total":26,"degraded_ratio":1087.6538461538462,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":426},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:22.574 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:22.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:22.918 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:22.937 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-22T08:37:22.431559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17508,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477520,"kb_used_data":183344,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281638000,"statfs":{"total":289910292480,"available":288397312000,"internally_reserved":0,"allocated":187744256,"data_stored":186994417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,8,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7864344,"num_objects":-1921,"num_object_clones":0,"num_object_copies":-3842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.017271"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20891,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442097+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:16.442097+0000","last_peered":"2024-04-22T08:37:16.442097+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:16.442097+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20065,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.430696+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:16.430696+0000","last_peered":"2024-04-22T08:37:16.430696+0000","last_clean":"2024-04-22T08:37:16.430696+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:16.430696+0000","last_undegraded":"2024-04-22T08:37:16.430696+0000","last_fullsized":"2024-04-22T08:37:16.430696+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.392589+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:16.392589+0000","last_peered":"2024-04-22T08:37:16.392589+0000","last_clean":"2024-04-22T08:37:16.392589+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:16.392589+0000","last_undegraded":"2024-04-22T08:37:16.392589+0000","last_fullsized":"2024-04-22T08:37:16.392589+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27607,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.428554+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:16.428554+0000","last_peered":"2024-04-22T08:37:16.428554+0000","last_clean":"2024-04-22T08:37:16.428554+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:16.428554+0000","last_undegraded":"2024-04-22T08:37:16.428554+0000","last_fullsized":"2024-04-22T08:37:16.428554+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19958,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:37:16.389395+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:16.389395+0000","last_peered":"2024-04-22T08:37:16.389395+0000","last_clean":"2024-04-22T08:37:16.389395+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:16.389395+0000","last_undegraded":"2024-04-22T08:37:16.389395+0000","last_fullsized":"2024-04-22T08:37:16.389395+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580386+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:35:02.580386+0000","last_peered":"2024-04-22T08:35:02.580386+0000","last_clean":"2024-04-22T08:35:02.580386+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:35:02.580386+0000","last_undegraded":"2024-04-22T08:35:02.580386+0000","last_fullsized":"2024-04-22T08:35:02.580386+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21072,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589598+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.589598+0000","last_peered":"2024-04-22T08:37:17.589598+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.589598+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":76,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589533+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.589533+0000","last_peered":"2024-04-22T08:37:17.589533+0000","last_clean":"2024-04-22T08:37:17.589533+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.589533+0000","last_undegraded":"2024-04-22T08:37:17.589533+0000","last_fullsized":"2024-04-22T08:37:17.589533+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580516+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:35:02.580516+0000","last_peered":"2024-04-22T08:35:02.580516+0000","last_clean":"2024-04-22T08:35:02.580516+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:35:02.580516+0000","last_undegraded":"2024-04-22T08:35:02.580516+0000","last_fullsized":"2024-04-22T08:35:02.580516+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20901,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.442495+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:16.442495+0000","last_peered":"2024-04-22T08:37:16.442495+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:16.442495+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580730+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:35:02.580730+0000","last_peered":"2024-04-22T08:35:02.580730+0000","last_clean":"2024-04-22T08:35:02.580730+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:35:02.580730+0000","last_undegraded":"2024-04-22T08:35:02.580730+0000","last_fullsized":"2024-04-22T08:35:02.580730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20427,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.411700+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:16.411700+0000","last_peered":"2024-04-22T08:37:16.411700+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:16.411700+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20737,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.716912+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:17.716912+0000","last_peered":"2024-04-22T08:37:17.716912+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:17.716912+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580815+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:35:02.580815+0000","last_peered":"2024-04-22T08:35:02.580815+0000","last_clean":"2024-04-22T08:35:02.580815+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:35:02.580815+0000","last_undegraded":"2024-04-22T08:35:02.580815+0000","last_fullsized":"2024-04-22T08:35:02.580815+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T08:35:02.580550+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:35:02.580550+0000","last_peered":"2024-04-22T08:35:02.580550+0000","last_clean":"2024-04-22T08:35:02.580550+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:35:02.580550+0000","last_undegraded":"2024-04-22T08:35:02.580550+0000","last_fullsized":"2024-04-22T08:35:02.580550+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20720,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:16.434505+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:16.434505+0000","last_peered":"2024-04-22T08:37:16.434505+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:16.434505+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17508,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117120,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.318},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.40100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487316,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884524,"statfs":{"total":96636764160,"available":96137752576,"internally_reserved":0,"allocated":405504,"data_stored":124448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313348,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511876,"kb_used_data":61116,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859964,"statfs":{"total":96636764160,"available":96112603136,"internally_reserved":0,"allocated":62582784,"data_stored":62356896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.309}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:25.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-04-22T08:37:24.432521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7864344,"num_objects":-1921,"num_object_clones":0,"num_object_copies":-3842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.017814"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":21860,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:22.714498+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:22.714498+0000","last_peered":"2024-04-22T08:37:22.714498+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:22.714498+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117120,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.318},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.40100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.69999999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313349,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:26.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:26.670 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:26.670 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:26.688 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-04-22T08:37:26.433337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3584000,"num_objects":-875,"num_object_clones":0,"num_object_copies":-1750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.847571"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":21860,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:22.714498+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:22.714498+0000","last_peered":"2024-04-22T08:37:22.714498+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:22.714498+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117120,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.318},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.40100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.69999999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313349,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:26.690 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 21860, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:22.714498+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:22.714498+0000', 'last_peered': '2024-04-22T08:37:22.714498+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:22.714498+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2839, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 870, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:26.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:26.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:27.067 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27538/26 objects degraded (105915.385%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1517207552,"bytes_avail":288393084928,"bytes_total":289910292480,"degraded_objects":27538,"degraded_total":26,"degraded_ratio":1059.1538461538462,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":98},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:27.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:27.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:27.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:27.395 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:27.413 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-04-22T08:37:26.433337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3584000,"num_objects":-875,"num_object_clones":0,"num_object_copies":-1750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.847571"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":21860,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:22.714498+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:22.714498+0000","last_peered":"2024-04-22T08:37:22.714498+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:22.714498+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2837}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18069,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117120,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.318},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.40100000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.69999999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313349,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.171,"15min":0.90900000000000003},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":6.1970000000000001,"5min":8,"15min":8},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.639,"5min":1.2130000000000001,"15min":0.94999999999999996},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":10.000999999999999,"5min":12.131,"15min":12.131},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:30.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:30.733 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:30.733 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:30.751 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-22T08:37:30.434494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.838229"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":22537,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:27.710290+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:27.710290+0000","last_peered":"2024-04-22T08:37:27.710290+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:27.710290+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117121,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.78300000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.627}]}]},{"osd":1,"up_from":33,"seq":141733920883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313350,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:30.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:31.096 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:31.097 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:31.117 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-22T08:37:30.434494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.838229"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":22537,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:27.710290+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:27.710290+0000","last_peered":"2024-04-22T08:37:27.710290+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:27.710290+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117121,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.78300000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.627}]}]},{"osd":1,"up_from":33,"seq":141733920883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313350,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:31.118 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:31.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 22537, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:27.710290+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:27.710290+0000', 'last_peered': '2024-04-22T08:37:27.710290+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:27.710290+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2501, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1208, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:31.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:31.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:31.484 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:31.484 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27200/26 objects degraded (104615.385%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":686,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1504624640,"bytes_avail":288405667840,"bytes_total":289910292480,"degraded_objects":27200,"degraded_total":26,"degraded_ratio":1046.1538461538462,"recovering_objects_per_sec":91,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:31.507 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:31.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:31.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:31.997 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-22T08:37:30.434494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.838229"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587192+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:17.587192+0000","last_peered":"2024-04-22T08:37:17.587192+0000","last_clean":"2024-04-22T08:37:17.587192+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:17.587192+0000","last_undegraded":"2024-04-22T08:37:17.587192+0000","last_fullsized":"2024-04-22T08:37:17.587192+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27598,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676000+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676000+0000","last_peered":"2024-04-22T08:37:18.676000+0000","last_clean":"2024-04-22T08:37:18.676000+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676000+0000","last_undegraded":"2024-04-22T08:37:18.676000+0000","last_fullsized":"2024-04-22T08:37:18.676000+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":40,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587496+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:17.587496+0000","last_peered":"2024-04-22T08:37:17.587496+0000","last_clean":"2024-04-22T08:37:17.587496+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:17.587496+0000","last_undegraded":"2024-04-22T08:37:17.587496+0000","last_fullsized":"2024-04-22T08:37:17.587496+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":22537,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:27.710290+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:27.710290+0000","last_peered":"2024-04-22T08:37:27.710290+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:27.710290+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18407,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117121,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.78300000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.627}]}]},{"osd":1,"up_from":33,"seq":141733920883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313350,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:34.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:35.339 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:35.339 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:35.357 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-22T08:37:34.436194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004634"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":23263,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:32.715794+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:32.715794+0000","last_peered":"2024-04-22T08:37:32.715794+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:32.715794+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117122,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.755}]}]},{"osd":1,"up_from":33,"seq":141733920884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313351,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:35.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:35.672 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:35.672 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-22T08:37:34.436194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004634"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":23263,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:32.715794+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:32.715794+0000","last_peered":"2024-04-22T08:37:32.715794+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:32.715794+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117122,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.755}]}]},{"osd":1,"up_from":33,"seq":141733920884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313351,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:35.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 23263, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:32.715794+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:32.715794+0000', 'last_peered': '2024-04-22T08:37:32.715794+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:32.715794+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1571, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:35.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:35.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:36.073 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26837/26 objects degraded (103219.231%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":691,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1508818944,"bytes_avail":288401473536,"bytes_total":289910292480,"degraded_objects":26837,"degraded_total":26,"degraded_ratio":1032.1923076923076,"recovering_objects_per_sec":105,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1262,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:36.092 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:36.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:36.411 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:36.411 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:36.428 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-22T08:37:34.436194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004634"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":23263,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:32.715794+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:32.715794+0000","last_peered":"2024-04-22T08:37:32.715794+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:32.715794+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18770,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117122,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.755}]}]},{"osd":1,"up_from":33,"seq":141733920884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313351,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:39.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:39.799 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:39.822 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-04-22T08:37:38.437405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004068"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24057,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:37.716058+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:37.716058+0000","last_peered":"2024-04-22T08:37:37.716058+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:37.716058+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117123,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313352,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:39.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:40.242 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:40.242 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:40.260 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-04-22T08:37:38.437405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004068"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24057,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:37.716058+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:37.716058+0000","last_peered":"2024-04-22T08:37:37.716058+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:37.716058+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117123,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313352,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:40.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:40.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 24057, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:37.716058+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:37.716058+0000', 'last_peered': '2024-04-22T08:37:37.716058+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:37.716058+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1741, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1968, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:40.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:40.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:40.667 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:40.667 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26440/26 objects degraded (101692.308%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":695,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1508818944,"bytes_avail":288401473536,"bytes_total":289910292480,"degraded_objects":26440,"degraded_total":26,"degraded_ratio":1016.9230769230769,"recovering_objects_per_sec":63,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":760,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:40.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:41.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:41.125 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:41.142 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-04-22T08:37:40.438277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1473456,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1290035,"kb_avail":281642064,"statfs":{"total":289910292480,"available":288401473536,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1320995950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004583"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19959,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.588065+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:17.588065+0000","last_peered":"2024-04-22T08:37:17.588065+0000","last_clean":"2024-04-22T08:37:17.588065+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:17.588065+0000","last_undegraded":"2024-04-22T08:37:17.588065+0000","last_fullsized":"2024-04-22T08:37:17.588065+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24057,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:37.716058+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:37.716058+0000","last_peered":"2024-04-22T08:37:37.716058+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:37.716058+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19167,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117123,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483228,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888612,"statfs":{"total":96636764160,"available":96141938688,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":494399180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313352,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:44.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:44.478 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:44.495 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-22T08:37:44.439303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477552,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637968,"statfs":{"total":289910292480,"available":288397279232,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24897,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:42.720215+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:42.720215+0000","last_peered":"2024-04-22T08:37:42.720215+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:42.720215+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.89400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487324,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884516,"statfs":{"total":96636764160,"available":96137744384,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.218},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313353,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:44.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:44.801 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:44.801 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:44.819 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-22T08:37:44.439303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477552,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637968,"statfs":{"total":289910292480,"available":288397279232,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24897,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:42.720215+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:42.720215+0000","last_peered":"2024-04-22T08:37:42.720215+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:42.720215+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.89400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487324,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884516,"statfs":{"total":96636764160,"available":96137744384,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.218},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313353,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:44.821 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 24897, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:42.720215+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:42.720215+0000', 'last_peered': '2024-04-22T08:37:42.720215+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:42.720215+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2388, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1319}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:44.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:45.211 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:45.211 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26020/26 objects degraded (100076.923%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":700,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1513013248,"bytes_avail":288397279232,"bytes_total":289910292480,"degraded_objects":26020,"degraded_total":26,"degraded_ratio":1000.7692307692308,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1180,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:45.230 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:45.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:45.528 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:45.528 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:45.546 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-22T08:37:44.439303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477552,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637968,"statfs":{"total":289910292480,"available":288397279232,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":24897,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:42.720215+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:42.720215+0000","last_peered":"2024-04-22T08:37:42.720215+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:42.720215+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19587,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.89400000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.54700000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487324,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884516,"statfs":{"total":96636764160,"available":96137744384,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.218},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.36199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313353,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:48.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:48.899 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:48.899 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:48.917 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-04-22T08:37:48.440406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":25919,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:47.719892+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:47.719892+0000","last_peered":"2024-04-22T08:37:47.719892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:47.719892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117125,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.70799999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313354,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:48.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:49.257 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:49.258 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:49.276 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-04-22T08:37:48.440406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":25919,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:47.719892+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:47.719892+0000","last_peered":"2024-04-22T08:37:47.719892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:47.719892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117125,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.70799999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313354,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:49.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:49.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:49.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:49.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 25919, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:47.719892+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:47.719892+0000', 'last_peered': '2024-04-22T08:37:47.719892+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:47.719892+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 810, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2899, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:49.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:49.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:49.657 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:49.657 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25509/26 objects degraded (98111.538%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":704,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1517207552,"bytes_avail":288393084928,"bytes_total":289910292480,"degraded_objects":25509,"degraded_total":26,"degraded_ratio":981.11538461538464,"recovering_objects_per_sec":110,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1328,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:49.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:49.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:49.999 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:50.027 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-04-22T08:37:48.440406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":25919,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:47.719892+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:47.719892+0000","last_peered":"2024-04-22T08:37:47.719892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:47.719892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117125,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.70799999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313354,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:53.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:53.351 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:53.351 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:53.372 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-04-22T08:37:52.441888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003610"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":25919,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:47.719892+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:47.719892+0000","last_peered":"2024-04-22T08:37:47.719892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:47.719892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67600000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313354,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:53.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:53.709 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:53.709 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:53.727 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-04-22T08:37:52.441888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481648,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281633872,"statfs":{"total":289910292480,"available":288393084928,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1329384558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003610"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20902,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589755+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:17.589755+0000","last_peered":"2024-04-22T08:37:17.589755+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:17.589755+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":25919,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:47.719892+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:47.719892+0000","last_peered":"2024-04-22T08:37:47.719892+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:47.719892+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20098,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67600000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491420,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880420,"statfs":{"total":96636764160,"available":96133550080,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":502787788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313354,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:53.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:53.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20902, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589755+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:17.589755+0000', 'last_peered': '2024-04-22T08:37:17.589755+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:17.589755+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 25919, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:47.719892+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:47.719892+0000', 'last_peered': '2024-04-22T08:37:47.719892+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:47.719892+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 810, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2899, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:53.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:53.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:54.101 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:54.101 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25509/26 objects degraded (98111.538%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1517207552,"bytes_avail":288393084928,"bytes_total":289910292480,"degraded_objects":25509,"degraded_total":26,"degraded_ratio":981.11538461538464,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:54.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:54.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:54.425 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:54.446 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-04-22T08:37:54.103756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.665131"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20903,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:18.602188+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:18.602188+0000","last_peered":"2024-04-22T08:37:18.602188+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:18.602188+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":27267,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:52.727874+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:52.727874+0000","last_peered":"2024-04-22T08:37:52.727874+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:52.727874+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67600000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313355,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:57.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:57.956 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-04-22T08:37:56.442903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":674,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002496"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20903,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:18.602188+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:18.602188+0000","last_peered":"2024-04-22T08:37:18.602188+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:18.602188+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":27267,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:52.727874+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:52.727874+0000","last_peered":"2024-04-22T08:37:52.727874+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:52.727874+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67600000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313355,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:57.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:58.325 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:58.345 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-04-22T08:37:56.442903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469360,"kb_used_data":183368,"kb_used_omap":12,"kb_used_meta":1285939,"kb_avail":281646160,"statfs":{"total":289910292480,"available":288405667840,"internally_reserved":0,"allocated":187768832,"data_stored":187004861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1316801646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":674,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002496"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20892,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.590526+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:17.590526+0000","last_peered":"2024-04-22T08:37:17.590526+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:17.590526+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20356,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587880+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:17.587880+0000","last_peered":"2024-04-22T08:37:17.587880+0000","last_clean":"2024-04-22T08:37:17.587880+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:17.587880+0000","last_undegraded":"2024-04-22T08:37:17.587880+0000","last_fullsized":"2024-04-22T08:37:17.587880+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20403,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.588387+0000","last_change":"2024-04-22T08:27:56.636327+0000","last_active":"2024-04-22T08:37:17.588387+0000","last_peered":"2024-04-22T08:37:17.588387+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:17.588387+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20153,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587414+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:17.587414+0000","last_peered":"2024-04-22T08:37:17.587414+0000","last_clean":"2024-04-22T08:37:17.587414+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:17.587414+0000","last_undegraded":"2024-04-22T08:37:17.587414+0000","last_fullsized":"2024-04-22T08:37:17.587414+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20066,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.589450+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:17.589450+0000","last_peered":"2024-04-22T08:37:17.589450+0000","last_clean":"2024-04-22T08:37:17.589450+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:17.589450+0000","last_undegraded":"2024-04-22T08:37:17.589450+0000","last_fullsized":"2024-04-22T08:37:17.589450+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27992,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585353+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:17.585353+0000","last_peered":"2024-04-22T08:37:17.585353+0000","last_clean":"2024-04-22T08:37:17.585353+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:17.585353+0000","last_undegraded":"2024-04-22T08:37:17.585353+0000","last_fullsized":"2024-04-22T08:37:17.585353+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27609,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585147+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:17.585147+0000","last_peered":"2024-04-22T08:37:17.585147+0000","last_clean":"2024-04-22T08:37:17.585147+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:17.585147+0000","last_undegraded":"2024-04-22T08:37:17.585147+0000","last_fullsized":"2024-04-22T08:37:17.585147+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597806+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:18.597806+0000","last_peered":"2024-04-22T08:37:18.597806+0000","last_clean":"2024-04-22T08:37:18.597806+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:18.597806+0000","last_undegraded":"2024-04-22T08:37:18.597806+0000","last_fullsized":"2024-04-22T08:37:18.597806+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587016+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:17.587016+0000","last_peered":"2024-04-22T08:37:17.587016+0000","last_clean":"2024-04-22T08:37:17.587016+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:17.587016+0000","last_undegraded":"2024-04-22T08:37:17.587016+0000","last_fullsized":"2024-04-22T08:37:17.587016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.587626+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:17.587626+0000","last_peered":"2024-04-22T08:37:17.587626+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:17.587626+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.597827+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:18.597827+0000","last_peered":"2024-04-22T08:37:18.597827+0000","last_clean":"2024-04-22T08:37:18.597827+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:18.597827+0000","last_undegraded":"2024-04-22T08:37:18.597827+0000","last_fullsized":"2024-04-22T08:37:18.597827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21073,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.717981+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:17.717981+0000","last_peered":"2024-04-22T08:37:17.717981+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:17.717981+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.717926+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:17.717926+0000","last_peered":"2024-04-22T08:37:17.717926+0000","last_clean":"2024-04-22T08:37:17.717926+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:17.717926+0000","last_undegraded":"2024-04-22T08:37:17.717926+0000","last_fullsized":"2024-04-22T08:37:17.717926+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.676437+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:18.676437+0000","last_peered":"2024-04-22T08:37:18.676437+0000","last_clean":"2024-04-22T08:37:18.676437+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:18.676437+0000","last_undegraded":"2024-04-22T08:37:18.676437+0000","last_fullsized":"2024-04-22T08:37:18.676437+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585762+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:17.585762+0000","last_peered":"2024-04-22T08:37:17.585762+0000","last_clean":"2024-04-22T08:37:17.585762+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:17.585762+0000","last_undegraded":"2024-04-22T08:37:17.585762+0000","last_fullsized":"2024-04-22T08:37:17.585762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27618,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.587768+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:17.587768+0000","last_peered":"2024-04-22T08:37:17.587768+0000","last_clean":"2024-04-22T08:37:17.587768+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:17.587768+0000","last_undegraded":"2024-04-22T08:37:17.587768+0000","last_fullsized":"2024-04-22T08:37:17.587768+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20903,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:18.602188+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:18.602188+0000","last_peered":"2024-04-22T08:37:18.602188+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:18.602188+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.586768+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:17.586768+0000","last_peered":"2024-04-22T08:37:17.586768+0000","last_clean":"2024-04-22T08:37:17.586768+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:17.586768+0000","last_undegraded":"2024-04-22T08:37:17.586768+0000","last_fullsized":"2024-04-22T08:37:17.586768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20428,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.591131+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:17.591131+0000","last_peered":"2024-04-22T08:37:17.591131+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:17.591131+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-22T08:37:18.600602+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:18.600602+0000","last_peered":"2024-04-22T08:37:18.600602+0000","last_clean":"2024-04-22T08:37:18.600602+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:18.600602+0000","last_undegraded":"2024-04-22T08:37:18.600602+0000","last_fullsized":"2024-04-22T08:37:18.600602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":27267,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:52.727874+0000","last_change":"2024-04-22T08:37:17.341659+0000","last_active":"2024-04-22T08:37:52.727874+0000","last_peered":"2024-04-22T08:37:52.727874+0000","last_clean":"2024-04-22T08:27:54.043046+0000","last_became_active":"2024-04-22T08:27:56.646268+0000","last_became_peered":"2024-04-22T08:27:56.646268+0000","last_unstale":"2024-04-22T08:37:52.727874+0000","last_undegraded":"2024-04-22T08:27:56.214004+0000","last_fullsized":"2024-04-22T08:27:56.089727+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585668+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:17.585668+0000","last_peered":"2024-04-22T08:37:17.585668+0000","last_clean":"2024-04-22T08:37:17.585668+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:17.585668+0000","last_undegraded":"2024-04-22T08:37:17.585668+0000","last_fullsized":"2024-04-22T08:37:17.585668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T08:37:17.585792+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:17.585792+0000","last_peered":"2024-04-22T08:37:17.585792+0000","last_clean":"2024-04-22T08:37:17.585792+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:17.585792+0000","last_undegraded":"2024-04-22T08:37:17.585792+0000","last_fullsized":"2024-04-22T08:37:17.585792+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20721,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:17.589768+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:17.589768+0000","last_peered":"2024-04-22T08:37:17.589768+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:17.589768+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20772,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186695680,"data_stored":186643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478328,"kb_used_data":121832,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893512,"statfs":{"total":96636764160,"available":96146956288,"internally_reserved":0,"allocated":124755968,"data_stored":124513073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.67600000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479132,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892708,"statfs":{"total":96636764160,"available":96146132992,"internally_reserved":0,"allocated":413696,"data_stored":129053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":490204876},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313355,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511900,"kb_used_data":61132,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859940,"statfs":{"total":96636764160,"available":96112578560,"internally_reserved":0,"allocated":62599168,"data_stored":62362735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":8021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20892, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.590526+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:17.590526+0000', 'last_peered': '2024-04-22T08:37:17.590526+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:17.590526+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 20403, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.588387+0000', 'last_change': '2024-04-22T08:27:56.636327+0000', 'last_active': '2024-04-22T08:37:17.588387+0000', 'last_peered': '2024-04-22T08:37:17.588387+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:37:17.588387+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3553, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:37:58.346 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.587626+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:17.587626+0000', 'last_peered': '2024-04-22T08:37:17.587626+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:17.587626+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21073, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.717981+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:17.717981+0000', 'last_peered': '2024-04-22T08:37:17.717981+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:17.717981+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20903, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:18.602188+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:18.602188+0000', 'last_peered': '2024-04-22T08:37:18.602188+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:18.602188+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20428, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.591131+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:17.591131+0000', 'last_peered': '2024-04-22T08:37:17.591131+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:17.591131+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'20038", 'reported_seq': 27267, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:52.727874+0000', 'last_change': '2024-04-22T08:37:17.341659+0000', 'last_active': '2024-04-22T08:37:52.727874+0000', 'last_peered': '2024-04-22T08:37:52.727874+0000', 'last_clean': '2024-04-22T08:27:54.043046+0000', 'last_became_active': '2024-04-22T08:27:56.646268+0000', 'last_became_peered': '2024-04-22T08:27:56.646268+0000', 'last_unstale': '2024-04-22T08:37:52.727874+0000', 'last_undegraded': '2024-04-22T08:27:56.214004+0000', 'last_fullsized': '2024-04-22T08:27:56.089727+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20036, 'num_write_kb': 40070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3573, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:37:58.347 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20721, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:17.589768+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:17.589768+0000', 'last_peered': '2024-04-22T08:37:17.589768+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:17.589768+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:37:58.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:37:59.095 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:59.095 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24311/26 objects degraded (93503.846%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":714,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1513054208,"bytes_avail":288397238272,"bytes_total":289910292480,"degraded_objects":24311,"degraded_total":26,"degraded_ratio":935.03846153846155,"recovering_objects_per_sec":149,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1198,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:37:59.116 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:37:59.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:37:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:37:59.499 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:37:59.519 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-04-22T08:37:58.443550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21296,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477592,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637928,"statfs":{"total":289910292480,"available":288397238272,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002374"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":21182,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:56.645946+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:37:56.645946+0000","last_peered":"2024-04-22T08:37:56.645946+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:37:56.645946+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3165}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21296,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117127,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.747}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.71899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487336,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884504,"statfs":{"total":96636764160,"available":96137732096,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53900000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:02.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:02.899 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:02.899 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:02.921 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-22T08:38:02.444333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477592,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637928,"statfs":{"total":289910292480,"available":288397238272,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.340576"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487336,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884504,"statfs":{"total":96636764160,"available":96137732096,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53900000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:02.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:03.245 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:03.245 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:03.265 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-22T08:38:02.444333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477592,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637928,"statfs":{"total":289910292480,"available":288397238272,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.340576"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487336,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884504,"statfs":{"total":96636764160,"available":96137732096,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53900000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:03.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:03.266 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20894, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111157+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:54.111157+0000', 'last_peered': '2024-04-22T08:37:54.111157+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:54.111157+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 23060, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:01.653017+0000', 'last_change': '2024-04-22T08:37:53.629002+0000', 'last_active': '2024-04-22T08:38:01.653017+0000', 'last_peered': '2024-04-22T08:38:01.653017+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:38:01.653017+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1530, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20685, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.110435+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:54.110435+0000', 'last_peered': '2024-04-22T08:37:54.110435+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:54.110435+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21075, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109679+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:54.109679+0000', 'last_peered': '2024-04-22T08:37:54.109679+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:54.109679+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20905, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109993+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:54.109993+0000', 'last_peered': '2024-04-22T08:37:54.109993+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:54.109993+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:03.267 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111232+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:54.111232+0000', 'last_peered': '2024-04-22T08:37:54.111232+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:54.111232+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:03.268 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:03.268 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20723, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.112145+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:54.112145+0000', 'last_peered': '2024-04-22T08:37:54.112145+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:54.112145+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:03.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:03.670 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:03.671 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23372/26 objects degraded (89892.308%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":718,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1513054208,"bytes_avail":288397238272,"bytes_total":289910292480,"degraded_objects":23372,"degraded_total":26,"degraded_ratio":898.92307692307691,"recovering_objects_per_sec":175,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:03.692 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:03.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:04.036 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:04.036 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:04.055 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-22T08:38:02.444333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1477592,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1294131,"kb_avail":281637928,"statfs":{"total":289910292480,"available":288397238272,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1325190254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.340576"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487336,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884504,"statfs":{"total":96636764160,"available":96137732096,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":498593484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53900000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313356,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:07.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:07.387 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:07.387 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:07.406 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-04-22T08:38:06.445136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1485784,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281629736,"statfs":{"total":289910292480,"available":288388849664,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1333578862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.331986"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495528,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":495099,"kb_avail":93876312,"statfs":{"total":96636764160,"available":96129343488,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":506982092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313357,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.252},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:07.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:07.705 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:07.705 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:07.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-04-22T08:38:06.445136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1485784,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281629736,"statfs":{"total":289910292480,"available":288388849664,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1333578862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.331986"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495528,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":495099,"kb_avail":93876312,"statfs":{"total":96636764160,"available":96129343488,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":506982092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313357,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.252},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20894, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111157+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:54.111157+0000', 'last_peered': '2024-04-22T08:37:54.111157+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:54.111157+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 23060, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:01.653017+0000', 'last_change': '2024-04-22T08:37:53.629002+0000', 'last_active': '2024-04-22T08:38:01.653017+0000', 'last_peered': '2024-04-22T08:38:01.653017+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:38:01.653017+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1530, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20685, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.110435+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:54.110435+0000', 'last_peered': '2024-04-22T08:37:54.110435+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:54.110435+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:07.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21075, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109679+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:54.109679+0000', 'last_peered': '2024-04-22T08:37:54.109679+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:54.109679+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20905, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109993+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:54.109993+0000', 'last_peered': '2024-04-22T08:37:54.109993+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:54.109993+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111232+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:54.111232+0000', 'last_peered': '2024-04-22T08:37:54.111232+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:54.111232+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:07.726 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20723, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.112145+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:54.112145+0000', 'last_peered': '2024-04-22T08:37:54.112145+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:54.112145+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:07.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:08.085 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23372/26 objects degraded (89892.308%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":723,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1521442816,"bytes_avail":288388849664,"bytes_total":289910292480,"degraded_objects":23372,"degraded_total":26,"degraded_ratio":898.92307692307691,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:08.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:08.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:08.386 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:08.387 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:08.403 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-04-22T08:38:06.445136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1485784,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281629736,"statfs":{"total":289910292480,"available":288388849664,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1333578862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.331986"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20894,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111157+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:54.111157+0000","last_peered":"2024-04-22T08:37:54.111157+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:54.111157+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":23060,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:01.653017+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:01.653017+0000","last_peered":"2024-04-22T08:38:01.653017+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:01.653017+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.441},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.53000000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495528,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":495099,"kb_avail":93876312,"statfs":{"total":96636764160,"available":96129343488,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":506982092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":21,"seq":90194313357,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.252},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:11.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:11.724 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:11.724 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:11.742 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-04-22T08:38:10.446263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23127,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491544,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623976,"statfs":{"total":289910292480,"available":288382951424,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002712"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":24844,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:06.644997+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:06.644997+0000","last_peered":"2024-04-22T08:38:06.644997+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:06.644997+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23127,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117129,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.45800000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501288,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870552,"statfs":{"total":96636764160,"available":96123445248,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.499},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313358,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:11.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:12.072 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-04-22T08:38:10.446263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23127,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491544,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623976,"statfs":{"total":289910292480,"available":288382951424,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002712"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":24844,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:06.644997+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:06.644997+0000","last_peered":"2024-04-22T08:38:06.644997+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:06.644997+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.110435+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:54.110435+0000","last_peered":"2024-04-22T08:37:54.110435+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:54.110435+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23127,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117129,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.3380000000000001,"15min":1.1339999999999999},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.419,"5min":15.6,"15min":15.6},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3999999999999999,"15min":1.0169999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.4409999999999998,"5min":18.32,"15min":18.32},"last":0.45800000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501288,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870552,"statfs":{"total":96636764160,"available":96123445248,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.499},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313358,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20895, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.117370+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:55.117370+0000', 'last_peered': '2024-04-22T08:37:55.117370+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:55.117370+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 24844, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:06.644997+0000', 'last_change': '2024-04-22T08:37:53.629002+0000', 'last_active': '2024-04-22T08:38:06.644997+0000', 'last_peered': '2024-04-22T08:38:06.644997+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:38:06.644997+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1334, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2422, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20685, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.110435+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:54.110435+0000', 'last_peered': '2024-04-22T08:37:54.110435+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:54.110435+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:12.074 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21075, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109679+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:54.109679+0000', 'last_peered': '2024-04-22T08:37:54.109679+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:54.109679+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20905, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109993+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:54.109993+0000', 'last_peered': '2024-04-22T08:37:54.109993+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:54.109993+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111232+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:54.111232+0000', 'last_peered': '2024-04-22T08:37:54.111232+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:54.111232+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:12.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20723, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.112145+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:54.112145+0000', 'last_peered': '2024-04-22T08:37:54.112145+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:54.112145+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:12.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:12.454 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:12.455 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22480/26 objects degraded (86461.538%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":727,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527341056,"bytes_avail":288382951424,"bytes_total":289910292480,"degraded_objects":22480,"degraded_total":26,"degraded_ratio":864.61538461538464,"recovering_objects_per_sec":152,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:12.475 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:12.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:12.808 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:12.829 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-04-22T08:38:12.446994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491544,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623976,"statfs":{"total":289910292480,"available":288382951424,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":25926,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:11.657980+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:11.657980+0000","last_peered":"2024-04-22T08:38:11.657980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:11.657980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.73199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501288,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870552,"statfs":{"total":96636764160,"available":96123445248,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.499},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313358,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.573,"15min":0.96899999999999997},"min":{"1min":0.191,"5min":0.16,"15min":0.16},"max":{"1min":10.435,"5min":11.368,"15min":11.368},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.079,"15min":1.0289999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.1740000000000004,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:15.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:16.152 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:16.152 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:16.170 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-04-22T08:38:14.447675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495640,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619880,"statfs":{"total":289910292480,"available":288378757120,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":25926,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:11.657980+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:11.657980+0000","last_peered":"2024-04-22T08:38:11.657980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:11.657980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.73199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505384,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866456,"statfs":{"total":96636764160,"available":96119250944,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313359,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:16.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:16.461 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:16.478 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-22T08:38:16.448472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495640,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619880,"statfs":{"total":289910292480,"available":288378757120,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":25926,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:11.657980+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:11.657980+0000","last_peered":"2024-04-22T08:38:11.657980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:11.657980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.73199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505384,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866456,"statfs":{"total":96636764160,"available":96119250944,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313359,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20895, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.117370+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:37:55.117370+0000', 'last_peered': '2024-04-22T08:37:55.117370+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:37:55.117370+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 25926, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:11.657980+0000', 'last_change': '2024-04-22T08:37:53.629002+0000', 'last_active': '2024-04-22T08:38:11.657980+0000', 'last_peered': '2024-04-22T08:38:11.657980+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:38:11.657980+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 793, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2963, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20686, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.114573+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:55.114573+0000', 'last_peered': '2024-04-22T08:37:55.114573+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:55.114573+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:16.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21075, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109679+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:54.109679+0000', 'last_peered': '2024-04-22T08:37:54.109679+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:54.109679+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20905, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.109993+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:54.109993+0000', 'last_peered': '2024-04-22T08:37:54.109993+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:54.109993+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.111232+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:54.111232+0000', 'last_peered': '2024-04-22T08:37:54.111232+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:54.111232+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:16.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20723, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:54.112145+0000', 'last_change': '2024-04-22T08:27:56.687635+0000', 'last_active': '2024-04-22T08:37:54.112145+0000', 'last_peered': '2024-04-22T08:37:54.112145+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:37:54.112145+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3601, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 288, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:16.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:16.860 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:16.860 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21939/26 objects degraded (84380.769%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":731,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531535360,"bytes_avail":288378757120,"bytes_total":289910292480,"degraded_objects":21939,"degraded_total":26,"degraded_ratio":843.80769230769215,"recovering_objects_per_sec":119,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:16.880 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:16.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:17.195 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:17.195 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:17.212 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-22T08:38:16.448472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495640,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619880,"statfs":{"total":289910292480,"available":288378757120,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20895,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117370+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:37:55.117370+0000","last_peered":"2024-04-22T08:37:55.117370+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:37:55.117370+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":25926,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:11.657980+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:11.657980+0000","last_peered":"2024-04-22T08:38:11.657980+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:11.657980+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20069,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.116106+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:55.116106+0000","last_peered":"2024-04-22T08:37:55.116106+0000","last_clean":"2024-04-22T08:37:55.116106+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:55.116106+0000","last_undegraded":"2024-04-22T08:37:55.116106+0000","last_fullsized":"2024-04-22T08:37:55.116106+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19962,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.108402+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:54.108402+0000","last_peered":"2024-04-22T08:37:54.108402+0000","last_clean":"2024-04-22T08:37:54.108402+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:54.108402+0000","last_undegraded":"2024-04-22T08:37:54.108402+0000","last_fullsized":"2024-04-22T08:37:54.108402+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21075,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109679+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:54.109679+0000","last_peered":"2024-04-22T08:37:54.109679+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:54.109679+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":79,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109221+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:54.109221+0000","last_peered":"2024-04-22T08:37:54.109221+0000","last_clean":"2024-04-22T08:37:54.109221+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:54.109221+0000","last_undegraded":"2024-04-22T08:37:54.109221+0000","last_fullsized":"2024-04-22T08:37:54.109221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20905,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.109993+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:54.109993+0000","last_peered":"2024-04-22T08:37:54.109993+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:54.109993+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.111232+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:54.111232+0000","last_peered":"2024-04-22T08:37:54.111232+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:54.111232+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20723,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:54.112145+0000","last_change":"2024-04-22T08:27:56.687635+0000","last_active":"2024-04-22T08:37:54.112145+0000","last_peered":"2024-04-22T08:37:54.112145+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:37:54.112145+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23668,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186691584,"data_stored":186642878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.73199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61799999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505384,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866456,"statfs":{"total":96636764160,"available":96119250944,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0429999999999999,"15min":0.76800000000000002},"min":{"1min":0.20999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.0629999999999997,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.1160000000000001,"15min":0.96999999999999997},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":4.2439999999999998,"5min":6.8860000000000001,"15min":6.8860000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Apr 22 08:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.004,"15min":1.339},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":3.5950000000000002,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0669999999999999,"15min":1.224},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":3.625,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313359,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:20.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:20.587 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:20.588 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:20.609 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-04-22T08:38:20.449762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487448,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628072,"statfs":{"total":289910292480,"available":288387145728,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002768"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20896,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:07.731813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:07.731813+0000","last_peered":"2024-04-22T08:38:07.731813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:07.731813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":27252,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:16.653769+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:16.653769+0000","last_peered":"2024-04-22T08:38:16.653769+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:16.653769+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3626,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20070,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:57.730710+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:57.730710+0000","last_peered":"2024-04-22T08:37:57.730710+0000","last_clean":"2024-04-22T08:37:57.730710+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:57.730710+0000","last_undegraded":"2024-04-22T08:37:57.730710+0000","last_fullsized":"2024-04-22T08:37:57.730710+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19963,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115750+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:55.115750+0000","last_peered":"2024-04-22T08:37:55.115750+0000","last_clean":"2024-04-22T08:37:55.115750+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:55.115750+0000","last_undegraded":"2024-04-22T08:37:55.115750+0000","last_fullsized":"2024-04-22T08:37:55.115750+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21076,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117481+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:55.117481+0000","last_peered":"2024-04-22T08:37:55.117481+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:55.117481+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":80,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115980+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:55.115980+0000","last_peered":"2024-04-22T08:37:55.115980+0000","last_clean":"2024-04-22T08:37:55.115980+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:55.115980+0000","last_undegraded":"2024-04-22T08:37:55.115980+0000","last_fullsized":"2024-04-22T08:37:55.115980+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20906,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.116573+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:55.116573+0000","last_peered":"2024-04-22T08:37:55.116573+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:55.116573+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20431,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117991+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:55.117991+0000","last_peered":"2024-04-22T08:37:55.117991+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:55.117991+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20805,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.732582+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:17.732582+0000","last_peered":"2024-04-22T08:38:17.732582+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:17.732582+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.497},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.54000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497192,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874648,"statfs":{"total":96636764160,"available":96127639552,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63}]}]},{"osd":0,"up_from":21,"seq":90194313360,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:20.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:20.935 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:20.954 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-04-22T08:38:20.449762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487448,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628072,"statfs":{"total":289910292480,"available":288387145728,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002768"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20896,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:07.731813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:07.731813+0000","last_peered":"2024-04-22T08:38:07.731813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:07.731813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":27252,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:16.653769+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:16.653769+0000","last_peered":"2024-04-22T08:38:16.653769+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:16.653769+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3626,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20070,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:57.730710+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:57.730710+0000","last_peered":"2024-04-22T08:37:57.730710+0000","last_clean":"2024-04-22T08:37:57.730710+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:57.730710+0000","last_undegraded":"2024-04-22T08:37:57.730710+0000","last_fullsized":"2024-04-22T08:37:57.730710+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19963,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115750+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:55.115750+0000","last_peered":"2024-04-22T08:37:55.115750+0000","last_clean":"2024-04-22T08:37:55.115750+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:55.115750+0000","last_undegraded":"2024-04-22T08:37:55.115750+0000","last_fullsized":"2024-04-22T08:37:55.115750+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21076,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117481+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:55.117481+0000","last_peered":"2024-04-22T08:37:55.117481+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:55.117481+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":80,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115980+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:55.115980+0000","last_peered":"2024-04-22T08:37:55.115980+0000","last_clean":"2024-04-22T08:37:55.115980+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:55.115980+0000","last_undegraded":"2024-04-22T08:37:55.115980+0000","last_fullsized":"2024-04-22T08:37:55.115980+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20906,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.116573+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:55.116573+0000","last_peered":"2024-04-22T08:37:55.116573+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:55.116573+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20431,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117991+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:55.117991+0000","last_peered":"2024-04-22T08:37:55.117991+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:55.117991+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20805,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.732582+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:17.732582+0000","last_peered":"2024-04-22T08:38:17.732582+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:17.732582+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.497},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.54000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497192,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874648,"statfs":{"total":96636764160,"available":96127639552,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63}]}]},{"osd":0,"up_from":21,"seq":90194313360,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20896, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:07.731813+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:07.731813+0000', 'last_peered': '2024-04-22T08:38:07.731813+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:07.731813+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'19864", 'reported_seq': 27252, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:16.653769+0000', 'last_change': '2024-04-22T08:37:53.629002+0000', 'last_active': '2024-04-22T08:38:16.653769+0000', 'last_peered': '2024-04-22T08:38:16.653769+0000', 'last_clean': '2024-04-22T08:27:54.071080+0000', 'last_became_active': '2024-04-22T08:27:56.621517+0000', 'last_became_peered': '2024-04-22T08:27:56.621517+0000', 'last_unstale': '2024-04-22T08:38:16.653769+0000', 'last_undegraded': '2024-04-22T08:27:56.201603+0000', 'last_fullsized': '2024-04-22T08:27:56.091031+0000', 'mapping_epoch': 34, 'log_start': "37'9800", 'ondisk_log_start': "37'9800", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "28'3232", 'last_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:44.400987+0000', 'objects_scrubbed': 3169, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000', 'scrub_duration': 1.556473761, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16691, 'num_write_kb': 27041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3626, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:20.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20686, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.114573+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:37:55.114573+0000', 'last_peered': '2024-04-22T08:37:55.114573+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:37:55.114573+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21076, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.117481+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:37:55.117481+0000', 'last_peered': '2024-04-22T08:37:55.117481+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:37:55.117481+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20906, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.116573+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:37:55.116573+0000', 'last_peered': '2024-04-22T08:37:55.116573+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:37:55.116573+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20431, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:37:55.117991+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:37:55.117991+0000', 'last_peered': '2024-04-22T08:37:55.117991+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:37:55.117991+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:20.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 20805, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.732582+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:17.732582+0000', 'last_peered': '2024-04-22T08:38:17.732582+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:17.732582+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 328, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:20.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:21.349 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:21.349 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21236/26 objects degraded (81676.923%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1523146752,"bytes_avail":288387145728,"bytes_total":289910292480,"degraded_objects":21236,"degraded_total":26,"degraded_ratio":816.76923076923083,"recovering_objects_per_sec":87,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":703,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:21.371 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:21.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:21.734 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:21.735 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:21.752 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-04-22T08:38:20.449762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487448,"kb_used_data":183412,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628072,"statfs":{"total":289910292480,"available":288387145728,"internally_reserved":0,"allocated":187813888,"data_stored":187021611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002768"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20896,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:07.731813+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:07.731813+0000","last_peered":"2024-04-22T08:38:07.731813+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:07.731813+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20358,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110454+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:37:54.110454+0000","last_peered":"2024-04-22T08:37:54.110454+0000","last_clean":"2024-04-22T08:37:54.110454+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:37:54.110454+0000","last_undegraded":"2024-04-22T08:37:54.110454+0000","last_fullsized":"2024-04-22T08:37:54.110454+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":27252,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:16.653769+0000","last_change":"2024-04-22T08:37:53.629002+0000","last_active":"2024-04-22T08:38:16.653769+0000","last_peered":"2024-04-22T08:38:16.653769+0000","last_clean":"2024-04-22T08:27:54.071080+0000","last_became_active":"2024-04-22T08:27:56.621517+0000","last_became_peered":"2024-04-22T08:27:56.621517+0000","last_unstale":"2024-04-22T08:38:16.653769+0000","last_undegraded":"2024-04-22T08:27:56.201603+0000","last_fullsized":"2024-04-22T08:27:56.091031+0000","mapping_epoch":34,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:09.357548+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3626,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109623+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:37:54.109623+0000","last_peered":"2024-04-22T08:37:54.109623+0000","last_clean":"2024-04-22T08:37:54.109623+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:37:54.109623+0000","last_undegraded":"2024-04-22T08:37:54.109623+0000","last_fullsized":"2024-04-22T08:37:54.109623+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20070,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:57.730710+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:37:57.730710+0000","last_peered":"2024-04-22T08:37:57.730710+0000","last_clean":"2024-04-22T08:37:57.730710+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:37:57.730710+0000","last_undegraded":"2024-04-22T08:37:57.730710+0000","last_fullsized":"2024-04-22T08:37:57.730710+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27994,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:37:54.105317+0000","last_peered":"2024-04-22T08:37:54.105317+0000","last_clean":"2024-04-22T08:37:54.105317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:37:54.105317+0000","last_undegraded":"2024-04-22T08:37:54.105317+0000","last_fullsized":"2024-04-22T08:37:54.105317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27611,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105145+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:37:54.105145+0000","last_peered":"2024-04-22T08:37:54.105145+0000","last_clean":"2024-04-22T08:37:54.105145+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:37:54.105145+0000","last_undegraded":"2024-04-22T08:37:54.105145+0000","last_fullsized":"2024-04-22T08:37:54.105145+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19963,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115750+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:37:55.115750+0000","last_peered":"2024-04-22T08:37:55.115750+0000","last_clean":"2024-04-22T08:37:55.115750+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:37:55.115750+0000","last_undegraded":"2024-04-22T08:37:55.115750+0000","last_fullsized":"2024-04-22T08:37:55.115750+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.120283+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:37:55.120283+0000","last_peered":"2024-04-22T08:37:55.120283+0000","last_clean":"2024-04-22T08:37:55.120283+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:37:55.120283+0000","last_undegraded":"2024-04-22T08:37:55.120283+0000","last_fullsized":"2024-04-22T08:37:55.120283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20686,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.114573+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:37:55.114573+0000","last_peered":"2024-04-22T08:37:55.114573+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:37:55.114573+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110120+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:37:54.110120+0000","last_peered":"2024-04-22T08:37:54.110120+0000","last_clean":"2024-04-22T08:37:54.110120+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:37:54.110120+0000","last_undegraded":"2024-04-22T08:37:54.110120+0000","last_fullsized":"2024-04-22T08:37:54.110120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21076,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117481+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:37:55.117481+0000","last_peered":"2024-04-22T08:37:55.117481+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:37:55.117481+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":80,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.115980+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:37:55.115980+0000","last_peered":"2024-04-22T08:37:55.115980+0000","last_clean":"2024-04-22T08:37:55.115980+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:37:55.115980+0000","last_undegraded":"2024-04-22T08:37:55.115980+0000","last_fullsized":"2024-04-22T08:37:55.115980+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27601,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.110194+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:37:54.110194+0000","last_peered":"2024-04-22T08:37:54.110194+0000","last_clean":"2024-04-22T08:37:54.110194+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:37:54.110194+0000","last_undegraded":"2024-04-22T08:37:54.110194+0000","last_fullsized":"2024-04-22T08:37:54.110194+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105496+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:37:54.105496+0000","last_peered":"2024-04-22T08:37:54.105496+0000","last_clean":"2024-04-22T08:37:54.105496+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:37:54.105496+0000","last_undegraded":"2024-04-22T08:37:54.105496+0000","last_fullsized":"2024-04-22T08:37:54.105496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.109892+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:37:54.109892+0000","last_peered":"2024-04-22T08:37:54.109892+0000","last_clean":"2024-04-22T08:37:54.109892+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:37:54.109892+0000","last_undegraded":"2024-04-22T08:37:54.109892+0000","last_fullsized":"2024-04-22T08:37:54.109892+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20906,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.116573+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:37:55.116573+0000","last_peered":"2024-04-22T08:37:55.116573+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:37:55.116573+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105479+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:37:54.105479+0000","last_peered":"2024-04-22T08:37:54.105479+0000","last_clean":"2024-04-22T08:37:54.105479+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:37:54.105479+0000","last_undegraded":"2024-04-22T08:37:54.105479+0000","last_fullsized":"2024-04-22T08:37:54.105479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20431,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:37:55.117991+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:37:55.117991+0000","last_peered":"2024-04-22T08:37:55.117991+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:37:55.117991+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.111525+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:37:54.111525+0000","last_peered":"2024-04-22T08:37:54.111525+0000","last_clean":"2024-04-22T08:37:54.111525+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:37:54.111525+0000","last_undegraded":"2024-04-22T08:37:54.111525+0000","last_fullsized":"2024-04-22T08:37:54.111525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20653,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-22T08:37:55.212497+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:37:55.212497+0000","last_peered":"2024-04-22T08:37:55.212497+0000","last_clean":"2024-04-22T08:37:55.212497+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:37:55.212497+0000","last_undegraded":"2024-04-22T08:37:55.212497+0000","last_fullsized":"2024-04-22T08:37:55.212497+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105676+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:37:54.105676+0000","last_peered":"2024-04-22T08:37:54.105676+0000","last_clean":"2024-04-22T08:37:54.105676+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:37:54.105676+0000","last_undegraded":"2024-04-22T08:37:54.105676+0000","last_fullsized":"2024-04-22T08:37:54.105676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-22T08:37:54.105620+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:37:54.105620+0000","last_peered":"2024-04-22T08:37:54.105620+0000","last_clean":"2024-04-22T08:37:54.105620+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:37:54.105620+0000","last_undegraded":"2024-04-22T08:37:54.105620+0000","last_fullsized":"2024-04-22T08:37:54.105620+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20805,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.732582+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:17.732582+0000","last_peered":"2024-04-22T08:38:17.732582+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:17.732582+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24371,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478344,"kb_used_data":121848,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893496,"statfs":{"total":96636764160,"available":96146939904,"internally_reserved":0,"allocated":124772352,"data_stored":124518862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.497},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.54000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497192,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874648,"statfs":{"total":96636764160,"available":96127639552,"internally_reserved":0,"allocated":425984,"data_stored":134225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63}]}]},{"osd":0,"up_from":21,"seq":90194313360,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511912,"kb_used_data":61148,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859928,"statfs":{"total":96636764160,"available":96112566272,"internally_reserved":0,"allocated":62615552,"data_stored":62368524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:24.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:25.101 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:25.102 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:25.121 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-04-22T08:38:24.450941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002468"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558436+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:17.558436+0000","last_peered":"2024-04-22T08:38:17.558436+0000","last_clean":"2024-04-22T08:38:17.558436+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:17.558436+0000","last_undegraded":"2024-04-22T08:38:17.558436+0000","last_fullsized":"2024-04-22T08:38:17.558436+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22099,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:22.739961+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:22.739961+0000","last_peered":"2024-04-22T08:38:22.739961+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:22.739961+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":974,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.65200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.56000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313361,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:25.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:25.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:25.446 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:25.466 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-04-22T08:38:24.450941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002468"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558436+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:17.558436+0000","last_peered":"2024-04-22T08:38:17.558436+0000","last_clean":"2024-04-22T08:38:17.558436+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:17.558436+0000","last_undegraded":"2024-04-22T08:38:17.558436+0000","last_fullsized":"2024-04-22T08:38:17.558436+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22099,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:22.739961+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:22.739961+0000","last_peered":"2024-04-22T08:38:22.739961+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:22.739961+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":974,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.65200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.56000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313361,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:25.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:25.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20432, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.549514+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:17.549514+0000', 'last_peered': '2024-04-22T08:38:17.549514+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:17.549514+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:25.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 22099, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:22.739961+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:22.739961+0000', 'last_peered': '2024-04-22T08:38:22.739961+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:22.739961+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2915, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 974, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:25.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:25.864 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:25.864 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20460/26 objects degraded (78692.308%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":740,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527382016,"bytes_avail":288382910464,"bytes_total":289910292480,"degraded_objects":20460,"degraded_total":26,"degraded_ratio":786.92307692307691,"recovering_objects_per_sec":184,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1479,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:25.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:25.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:26.188 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:26.189 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:26.207 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-04-22T08:38:24.450941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1479,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002468"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558436+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:17.558436+0000","last_peered":"2024-04-22T08:38:17.558436+0000","last_clean":"2024-04-22T08:38:17.558436+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:17.558436+0000","last_undegraded":"2024-04-22T08:38:17.558436+0000","last_fullsized":"2024-04-22T08:38:17.558436+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22099,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:22.739961+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:22.739961+0000","last_peered":"2024-04-22T08:38:22.739961+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:22.739961+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":974,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25147,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:37:20 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.073,"15min":1.0209999999999999},"min":{"1min":0.191,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":6.3860000000000001,"5min":6.3860000000000001,"15min":13.635},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.871,"15min":0.80900000000000005},"min":{"1min":0.25800000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":6.431,"5min":6.7670000000000003,"15min":13.721},"last":0.65200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.56000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313361,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:29.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:29.527 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:29.544 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-04-22T08:38:28.452279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003099"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22765,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:27.735131+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:27.735131+0000","last_peered":"2024-04-22T08:38:27.735131+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:27.735131+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.75900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313362,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:29.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:29.851 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:29.870 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-04-22T08:38:28.452279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003099"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22765,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:27.735131+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:27.735131+0000","last_peered":"2024-04-22T08:38:27.735131+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:27.735131+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.75900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313362,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:29.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:29.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:29.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:29.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20432, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.549514+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:17.549514+0000', 'last_peered': '2024-04-22T08:38:17.549514+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:17.549514+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:29.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 22765, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:27.735131+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:27.735131+0000', 'last_peered': '2024-04-22T08:38:27.735131+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:27.735131+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1307, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:29.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:30.250 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:30.250 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20127/26 objects degraded (77411.538%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":745,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531576320,"bytes_avail":288378716160,"bytes_total":289910292480,"degraded_objects":20127,"degraded_total":26,"degraded_ratio":774.11538461538464,"recovering_objects_per_sec":110,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:30.271 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:30.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:30.568 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:30.569 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:30.586 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-04-22T08:38:30.452973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.892639"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22765,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:27.735131+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:27.735131+0000","last_peered":"2024-04-22T08:38:27.735131+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:27.735131+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.75900000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313362,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:33.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:33.915 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:33.916 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:33.938 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-22T08:38:32.453396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003634"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22765,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:27.735131+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:27.735131+0000","last_peered":"2024-04-22T08:38:27.735131+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:27.735131+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.378}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.29699999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313362,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:33.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:34.243 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:34.243 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:34.260 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-22T08:38:32.453396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003634"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":22765,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:27.735131+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:27.735131+0000","last_peered":"2024-04-22T08:38:27.735131+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:27.735131+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25480,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.378}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.29699999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.65700000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313362,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.167,"15min":0.97999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19800000000000001,"15min":0.18099999999999999},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":3.0409999999999999,"5min":1.671,"15min":1.101},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":11.141999999999999,"5min":12.131,"15min":12.131},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:34.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:34.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:34.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20432, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.549514+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:17.549514+0000', 'last_peered': '2024-04-22T08:38:17.549514+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:17.549514+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:34.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:34.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 22765, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:27.735131+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:27.735131+0000', 'last_peered': '2024-04-22T08:38:27.735131+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:27.735131+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1307, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:34.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:34.630 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:34.630 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20127/26 objects degraded (77411.538%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":749,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531576320,"bytes_avail":288378716160,"bytes_total":289910292480,"degraded_objects":20127,"degraded_total":26,"degraded_ratio":774.11538461538464,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:34.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:34.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:34.957 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:34.957 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:34.975 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-04-22T08:38:34.454115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25845,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003869"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":23495,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:32.744767+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:32.744767+0000","last_peered":"2024-04-22T08:38:32.744767+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:32.744767+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1672,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25845,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.378}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.29699999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":1.0469999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313363,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.871},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.94299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:37.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:38.295 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:38.296 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:38.318 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-04-22T08:38:36.454668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25845,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20432,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.549514+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:17.549514+0000","last_peered":"2024-04-22T08:38:17.549514+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:17.549514+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":23495,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:32.744767+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:32.744767+0000","last_peered":"2024-04-22T08:38:32.744767+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:32.744767+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1672,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25845,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.378}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.29699999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":1.0469999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313363,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.871},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.94299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:38.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:38.659 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:38.659 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:38.680 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-04-22T08:38:38.455234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1076,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003561"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":24251,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:37.742720+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:37.742720+0000","last_peered":"2024-04-22T08:38:37.742720+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:37.742720+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117135,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.67000000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.504},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61299999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920897,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313364,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:38.681 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:38.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20433, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:18.566967+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:18.566967+0000', 'last_peered': '2024-04-22T08:38:18.566967+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:18.566967+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:38.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:38.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 24251, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:37.742720+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:37.742720+0000', 'last_peered': '2024-04-22T08:38:37.742720+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:37.742720+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1839, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2050, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:38.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:39.059 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:39.059 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19384/26 objects degraded (74553.846%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535770624,"bytes_avail":288374521856,"bytes_total":289910292480,"degraded_objects":19384,"degraded_total":26,"degraded_ratio":745.53846153846155,"recovering_objects_per_sec":89,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1076,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-22T08:36:56.409059+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:39.078 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:39.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:39.379 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:39.380 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:39.398 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-04-22T08:38:38.455234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1076,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003561"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":24251,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:37.742720+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:37.742720+0000","last_peered":"2024-04-22T08:38:37.742720+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:37.742720+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117135,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.67000000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.504},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.61299999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920897,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313364,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:42.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:42.721 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:42.722 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:42.740 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-22T08:38:42.456253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":24251,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:37.742720+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:37.742720+0000","last_peered":"2024-04-22T08:38:37.742720+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:37.742720+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920897,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313364,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:42.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:43.066 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:43.067 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:43.086 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-22T08:38:42.456253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":24251,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:37.742720+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:37.742720+0000","last_peered":"2024-04-22T08:38:37.742720+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:37.742720+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920897,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313364,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20433, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:18.566967+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:18.566967+0000', 'last_peered': '2024-04-22T08:38:18.566967+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:18.566967+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:43.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 24251, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:37.742720+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:37.742720+0000', 'last_peered': '2024-04-22T08:38:37.742720+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:37.742720+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1839, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2050, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:43.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:43.450 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:43.450 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19384/26 objects degraded (74553.846%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":758,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535770624,"bytes_avail":288374521856,"bytes_total":289910292480,"degraded_objects":19384,"degraded_total":26,"degraded_ratio":745.53846153846155,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":743,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:43.469 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:43.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:43.763 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:43.763 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:43.782 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-22T08:38:42.456253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499776,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615744,"statfs":{"total":289910292480,"available":288374521856,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":24251,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:37.742720+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:37.742720+0000","last_peered":"2024-04-22T08:38:37.742720+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:37.742720+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26223,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920897,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509484,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862356,"statfs":{"total":96636764160,"available":96115052544,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60399999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313364,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:46.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:47.133 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:47.134 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:47.151 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-04-22T08:38:46.457587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487488,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628032,"statfs":{"total":289910292480,"available":288387104768,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":25127,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:42.748644+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:42.748644+0000","last_peered":"2024-04-22T08:38:42.748644+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:42.748644+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2488,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497196,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874644,"statfs":{"total":96636764160,"available":96127635456,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313365,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:47.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:47.469 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:47.470 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:47.489 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-04-22T08:38:46.457587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487488,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628032,"statfs":{"total":289910292480,"available":288387104768,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":25127,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:42.748644+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:42.748644+0000","last_peered":"2024-04-22T08:38:42.748644+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:42.748644+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2488,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497196,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874644,"statfs":{"total":96636764160,"available":96127635456,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313365,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:47.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:47.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:47.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20433, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:18.566967+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:18.566967+0000', 'last_peered': '2024-04-22T08:38:18.566967+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:18.566967+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:47.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:47.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 25127, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:42.748644+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:42.748644+0000', 'last_peered': '2024-04-22T08:38:42.748644+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:42.748644+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1401, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2488, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:47.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:47.859 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18946/26 objects degraded (72869.231%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":762,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1523187712,"bytes_avail":288387104768,"bytes_total":289910292480,"degraded_objects":18946,"degraded_total":26,"degraded_ratio":728.69230769230785,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:47.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:47.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:48.181 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:48.181 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:48.199 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-04-22T08:38:46.457587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487488,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281628032,"statfs":{"total":289910292480,"available":288387104768,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558129+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:17.558129+0000","last_peered":"2024-04-22T08:38:17.558129+0000","last_clean":"2024-04-22T08:38:17.558129+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:17.558129+0000","last_undegraded":"2024-04-22T08:38:17.558129+0000","last_fullsized":"2024-04-22T08:38:17.558129+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":25127,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:42.748644+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:42.748644+0000","last_peered":"2024-04-22T08:38:42.748644+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:42.748644+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2488,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26661,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.437}]}]},{"osd":1,"up_from":33,"seq":141733920898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497196,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874644,"statfs":{"total":96636764160,"available":96127635456,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313365,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:51.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:51.532 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-04-22T08:38:50.459112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":26177,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:47.752531+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:47.752531+0000","last_peered":"2024-04-22T08:38:47.752531+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:47.752531+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117137,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.65700000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313366,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.309},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:51.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:51.869 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:51.869 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:51.888 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-04-22T08:38:50.459112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":26177,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:47.752531+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:47.752531+0000","last_peered":"2024-04-22T08:38:47.752531+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:47.752531+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117137,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.65700000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313366,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.309},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:51.889 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20433, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:18.566967+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:18.566967+0000', 'last_peered': '2024-04-22T08:38:18.566967+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:18.566967+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:51.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 26177, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:47.752531+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:47.752531+0000', 'last_peered': '2024-04-22T08:38:47.752531+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:47.752531+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 876, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3013, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:51.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:52.309 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18421/26 objects degraded (70850.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":767,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527382016,"bytes_avail":288382910464,"bytes_total":289910292480,"degraded_objects":18421,"degraded_total":26,"degraded_ratio":708.5,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1341,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:52.333 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:52.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:52.633 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:52.633 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:52.650 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-04-22T08:38:52.459812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491584,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623936,"statfs":{"total":289910292480,"available":288382910464,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004137"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":26177,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:47.752531+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:47.752531+0000","last_peered":"2024-04-22T08:38:47.752531+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:47.752531+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3013,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27186,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.309}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.28699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501292,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870548,"statfs":{"total":96636764160,"available":96123441152,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313366,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.248}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.309},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:55.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:55.988 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:56.006 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-04-22T08:38:54.460418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004164"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":27571,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:52.742104+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:52.742104+0000","last_peered":"2024-04-22T08:38:52.742104+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:52.742104+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.309}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.28699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313367,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:56.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:56.342 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:56.344 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:56.361 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-04-22T08:38:56.173684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.716966"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":27571,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:52.742104+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:52.742104+0000","last_peered":"2024-04-22T08:38:52.742104+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:52.742104+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.309}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.28699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313367,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20897, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.552904+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:17.552904+0000', 'last_peered': '2024-04-22T08:38:17.552904+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:17.552904+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20688, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546897+0000', 'last_change': '2024-04-22T08:27:56.667684+0000', 'last_active': '2024-04-22T08:38:17.546897+0000', 'last_peered': '2024-04-22T08:38:17.546897+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:17.546897+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3725, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21077, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.546220+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:17.546220+0000', 'last_peered': '2024-04-22T08:38:17.546220+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:17.546220+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:38:56.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20907, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:17.551569+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:17.551569+0000', 'last_peered': '2024-04-22T08:38:17.551569+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:17.551569+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:56.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:38:56.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20433, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:18.566967+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:18.566967+0000', 'last_peered': '2024-04-22T08:38:18.566967+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:18.566967+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:56.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-04-22T08:38:56.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'20002", 'reported_seq': 27571, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:52.742104+0000', 'last_change': '2024-04-22T08:38:17.200675+0000', 'last_active': '2024-04-22T08:38:52.742104+0000', 'last_peered': '2024-04-22T08:38:52.742104+0000', 'last_clean': '2024-04-22T08:27:54.081188+0000', 'last_became_active': '2024-04-22T08:27:56.680501+0000', 'last_became_peered': '2024-04-22T08:27:56.680501+0000', 'last_unstale': '2024-04-22T08:38:52.742104+0000', 'last_undegraded': '2024-04-22T08:27:56.223120+0000', 'last_fullsized': '2024-04-22T08:27:56.085668+0000', 'mapping_epoch': 34, 'log_start': "37'10000", 'ondisk_log_start': "37'10000", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 2649, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 179, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19999, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3710, 'num_bytes_recovered': 4713, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:38:56.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:38:56.807 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:56.808 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17724/26 objects degraded (68169.231%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":771,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531576320,"bytes_avail":288378716160,"bytes_total":289910292480,"degraded_objects":17724,"degraded_total":26,"degraded_ratio":681.69230769230785,"recovering_objects_per_sec":138,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1660,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:38:56.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:38:56.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:38:57.175 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:38:57.175 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:38:57.204 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-04-22T08:38:56.461082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":183448,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":187850752,"data_stored":187036977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003495"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20897,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.552904+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:17.552904+0000","last_peered":"2024-04-22T08:38:17.552904+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:17.552904+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20360,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545957+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:17.545957+0000","last_peered":"2024-04-22T08:38:17.545957+0000","last_clean":"2024-04-22T08:38:17.545957+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:17.545957+0000","last_undegraded":"2024-04-22T08:38:17.545957+0000","last_fullsized":"2024-04-22T08:38:17.545957+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20413,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.712890+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:18.712890+0000","last_peered":"2024-04-22T08:38:18.712890+0000","last_clean":"2024-04-22T08:38:18.712890+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:18.712890+0000","last_undegraded":"2024-04-22T08:38:18.712890+0000","last_fullsized":"2024-04-22T08:38:18.712890+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.570099+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:18.570099+0000","last_peered":"2024-04-22T08:38:18.570099+0000","last_clean":"2024-04-22T08:38:18.570099+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:18.570099+0000","last_undegraded":"2024-04-22T08:38:18.570099+0000","last_fullsized":"2024-04-22T08:38:18.570099+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20071,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.546218+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:17.546218+0000","last_peered":"2024-04-22T08:38:17.546218+0000","last_clean":"2024-04-22T08:38:17.546218+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:17.546218+0000","last_undegraded":"2024-04-22T08:38:17.546218+0000","last_fullsized":"2024-04-22T08:38:17.546218+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27996,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556444+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:17.556444+0000","last_peered":"2024-04-22T08:38:17.556444+0000","last_clean":"2024-04-22T08:38:17.556444+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:17.556444+0000","last_undegraded":"2024-04-22T08:38:17.556444+0000","last_fullsized":"2024-04-22T08:38:17.556444+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27613,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556286+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:17.556286+0000","last_peered":"2024-04-22T08:38:17.556286+0000","last_clean":"2024-04-22T08:38:17.556286+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:17.556286+0000","last_undegraded":"2024-04-22T08:38:17.556286+0000","last_fullsized":"2024-04-22T08:38:17.556286+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19964,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544747+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:17.544747+0000","last_peered":"2024-04-22T08:38:17.544747+0000","last_clean":"2024-04-22T08:38:17.544747+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:17.544747+0000","last_undegraded":"2024-04-22T08:38:17.544747+0000","last_fullsized":"2024-04-22T08:38:17.544747+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.556302+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:17.556302+0000","last_peered":"2024-04-22T08:38:17.556302+0000","last_clean":"2024-04-22T08:38:17.556302+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:17.556302+0000","last_undegraded":"2024-04-22T08:38:17.556302+0000","last_fullsized":"2024-04-22T08:38:17.556302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20688,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546897+0000","last_change":"2024-04-22T08:27:56.667684+0000","last_active":"2024-04-22T08:38:17.546897+0000","last_peered":"2024-04-22T08:38:17.546897+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:17.546897+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544137+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:17.544137+0000","last_peered":"2024-04-22T08:38:17.544137+0000","last_clean":"2024-04-22T08:38:17.544137+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:17.544137+0000","last_undegraded":"2024-04-22T08:38:17.544137+0000","last_fullsized":"2024-04-22T08:38:17.544137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21077,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.546220+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:17.546220+0000","last_peered":"2024-04-22T08:38:17.546220+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:17.546220+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545116+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:17.545116+0000","last_peered":"2024-04-22T08:38:17.545116+0000","last_clean":"2024-04-22T08:38:17.545116+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:17.545116+0000","last_undegraded":"2024-04-22T08:38:17.545116+0000","last_fullsized":"2024-04-22T08:38:17.545116+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.544148+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:17.544148+0000","last_peered":"2024-04-22T08:38:17.544148+0000","last_clean":"2024-04-22T08:38:17.544148+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:17.544148+0000","last_undegraded":"2024-04-22T08:38:17.544148+0000","last_fullsized":"2024-04-22T08:38:17.544148+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.563635+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:18.563635+0000","last_peered":"2024-04-22T08:38:18.563635+0000","last_clean":"2024-04-22T08:38:18.563635+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:18.563635+0000","last_undegraded":"2024-04-22T08:38:18.563635+0000","last_fullsized":"2024-04-22T08:38:18.563635+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27622,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.543851+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:17.543851+0000","last_peered":"2024-04-22T08:38:17.543851+0000","last_clean":"2024-04-22T08:38:17.543851+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:17.543851+0000","last_undegraded":"2024-04-22T08:38:17.543851+0000","last_fullsized":"2024-04-22T08:38:17.543851+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20907,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:17.551569+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:17.551569+0000","last_peered":"2024-04-22T08:38:17.551569+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:17.551569+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.566971+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:18.566971+0000","last_peered":"2024-04-22T08:38:18.566971+0000","last_clean":"2024-04-22T08:38:18.566971+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:18.566971+0000","last_undegraded":"2024-04-22T08:38:18.566971+0000","last_fullsized":"2024-04-22T08:38:18.566971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20433,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:18.566967+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:18.566967+0000","last_peered":"2024-04-22T08:38:18.566967+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:18.566967+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.545375+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:17.545375+0000","last_peered":"2024-04-22T08:38:17.545375+0000","last_clean":"2024-04-22T08:38:17.545375+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:17.545375+0000","last_undegraded":"2024-04-22T08:38:17.545375+0000","last_fullsized":"2024-04-22T08:38:17.545375+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20655,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-22T08:38:17.558661+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:17.558661+0000","last_peered":"2024-04-22T08:38:17.558661+0000","last_clean":"2024-04-22T08:38:17.558661+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:17.558661+0000","last_undegraded":"2024-04-22T08:38:17.558661+0000","last_fullsized":"2024-04-22T08:38:17.558661+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565573+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:18.565573+0000","last_peered":"2024-04-22T08:38:18.565573+0000","last_clean":"2024-04-22T08:38:18.565573+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:18.565573+0000","last_undegraded":"2024-04-22T08:38:18.565573+0000","last_fullsized":"2024-04-22T08:38:18.565573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.565706+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:18.565706+0000","last_peered":"2024-04-22T08:38:18.565706+0000","last_clean":"2024-04-22T08:38:18.565706+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:18.565706+0000","last_undegraded":"2024-04-22T08:38:18.565706+0000","last_fullsized":"2024-04-22T08:38:18.565706+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":27571,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:52.742104+0000","last_change":"2024-04-22T08:38:17.200675+0000","last_active":"2024-04-22T08:38:52.742104+0000","last_peered":"2024-04-22T08:38:52.742104+0000","last_clean":"2024-04-22T08:27:54.081188+0000","last_became_active":"2024-04-22T08:27:56.680501+0000","last_became_peered":"2024-04-22T08:27:56.680501+0000","last_unstale":"2024-04-22T08:38:52.742104+0000","last_undegraded":"2024-04-22T08:27:56.223120+0000","last_fullsized":"2024-04-22T08:27:56.085668+0000","mapping_epoch":34,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27883,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186679296,"data_stored":186641027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478360,"kb_used_data":121864,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893480,"statfs":{"total":96636764160,"available":96146923520,"internally_reserved":0,"allocated":124788736,"data_stored":124524601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.309}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.28699999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505388,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866452,"statfs":{"total":96636764160,"available":96119246848,"internally_reserved":0,"allocated":430080,"data_stored":138113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313367,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511932,"kb_used_data":61164,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859908,"statfs":{"total":96636764160,"available":96112545792,"internally_reserved":0,"allocated":62631936,"data_stored":62374263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:00.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:00.608 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:00.609 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:00.630 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-04-22T08:39:00.462094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499800,"kb_used_data":183472,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615720,"statfs":{"total":289910292480,"available":288374497280,"internally_reserved":0,"allocated":187875328,"data_stored":187047142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002281"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20362,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.175585+0000","last_peered":"2024-04-22T08:38:56.175585+0000","last_clean":"2024-04-22T08:38:56.175585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.175585+0000","last_undegraded":"2024-04-22T08:38:56.175585+0000","last_fullsized":"2024-04-22T08:38:56.175585+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:21.659229+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:21.659229+0000","last_peered":"2024-04-22T08:38:21.659229+0000","last_clean":"2024-04-22T08:38:21.659229+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:21.659229+0000","last_undegraded":"2024-04-22T08:38:21.659229+0000","last_fullsized":"2024-04-22T08:38:21.659229+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20874,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.668994+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:38:56.668994+0000","last_peered":"2024-04-22T08:38:56.668994+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:56.668994+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569282+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:18.569282+0000","last_peered":"2024-04-22T08:38:18.569282+0000","last_clean":"2024-04-22T08:38:18.569282+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:18.569282+0000","last_undegraded":"2024-04-22T08:38:18.569282+0000","last_fullsized":"2024-04-22T08:38:18.569282+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27604,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569476+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:18.569476+0000","last_peered":"2024-04-22T08:38:18.569476+0000","last_clean":"2024-04-22T08:38:18.569476+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:18.569476+0000","last_undegraded":"2024-04-22T08:38:18.569476+0000","last_fullsized":"2024-04-22T08:38:18.569476+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27623,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.568128+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:18.568128+0000","last_peered":"2024-04-22T08:38:18.568128+0000","last_clean":"2024-04-22T08:38:18.568128+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:18.568128+0000","last_undegraded":"2024-04-22T08:38:18.568128+0000","last_fullsized":"2024-04-22T08:38:18.568128+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569079+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:18.569079+0000","last_peered":"2024-04-22T08:38:18.569079+0000","last_clean":"2024-04-22T08:38:18.569079+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:18.569079+0000","last_undegraded":"2024-04-22T08:38:18.569079+0000","last_fullsized":"2024-04-22T08:38:18.569079+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117139,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478364,"kb_used_data":121872,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893476,"statfs":{"total":96636764160,"available":96146919424,"internally_reserved":0,"allocated":124796928,"data_stored":124527456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509488,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862352,"statfs":{"total":96636764160,"available":96115048448,"internally_reserved":0,"allocated":430080,"data_stored":139734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313368,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:00.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:00.961 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:00.962 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:00.983 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-04-22T08:39:00.462094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499800,"kb_used_data":183472,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615720,"statfs":{"total":289910292480,"available":288374497280,"internally_reserved":0,"allocated":187875328,"data_stored":187047142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002281"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20362,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.175585+0000","last_peered":"2024-04-22T08:38:56.175585+0000","last_clean":"2024-04-22T08:38:56.175585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.175585+0000","last_undegraded":"2024-04-22T08:38:56.175585+0000","last_fullsized":"2024-04-22T08:38:56.175585+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:21.659229+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:21.659229+0000","last_peered":"2024-04-22T08:38:21.659229+0000","last_clean":"2024-04-22T08:38:21.659229+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:21.659229+0000","last_undegraded":"2024-04-22T08:38:21.659229+0000","last_fullsized":"2024-04-22T08:38:21.659229+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20874,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.668994+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:38:56.668994+0000","last_peered":"2024-04-22T08:38:56.668994+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:56.668994+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569282+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:18.569282+0000","last_peered":"2024-04-22T08:38:18.569282+0000","last_clean":"2024-04-22T08:38:18.569282+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:18.569282+0000","last_undegraded":"2024-04-22T08:38:18.569282+0000","last_fullsized":"2024-04-22T08:38:18.569282+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27604,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569476+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:18.569476+0000","last_peered":"2024-04-22T08:38:18.569476+0000","last_clean":"2024-04-22T08:38:18.569476+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:18.569476+0000","last_undegraded":"2024-04-22T08:38:18.569476+0000","last_fullsized":"2024-04-22T08:38:18.569476+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27623,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.568128+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:18.568128+0000","last_peered":"2024-04-22T08:38:18.568128+0000","last_clean":"2024-04-22T08:38:18.568128+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:18.568128+0000","last_undegraded":"2024-04-22T08:38:18.568128+0000","last_fullsized":"2024-04-22T08:38:18.568128+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569079+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:18.569079+0000","last_peered":"2024-04-22T08:38:18.569079+0000","last_clean":"2024-04-22T08:38:18.569079+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:18.569079+0000","last_undegraded":"2024-04-22T08:38:18.569079+0000","last_fullsized":"2024-04-22T08:38:18.569079+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117139,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478364,"kb_used_data":121872,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893476,"statfs":{"total":96636764160,"available":96146919424,"internally_reserved":0,"allocated":124796928,"data_stored":124527456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509488,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862352,"statfs":{"total":96636764160,"available":96115048448,"internally_reserved":0,"allocated":430080,"data_stored":139734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313368,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20874, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.668994+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:38:56.668994+0000', 'last_peered': '2024-04-22T08:38:56.668994+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:38:56.668994+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3633, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 262, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:00.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:00.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:00.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:00.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:00.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:00.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:01.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:01.389 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17453/26 objects degraded (67126.923%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":776,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535795200,"bytes_avail":288374497280,"bytes_total":289910292480,"degraded_objects":17453,"degraded_total":26,"degraded_ratio":671.26923076923083,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:01.409 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:01.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:01.716 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:01.717 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:01.735 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-04-22T08:39:00.462094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499800,"kb_used_data":183472,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615720,"statfs":{"total":289910292480,"available":288374497280,"internally_reserved":0,"allocated":187875328,"data_stored":187047142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002281"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20362,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175585+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.175585+0000","last_peered":"2024-04-22T08:38:56.175585+0000","last_clean":"2024-04-22T08:38:56.175585+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.175585+0000","last_undegraded":"2024-04-22T08:38:56.175585+0000","last_fullsized":"2024-04-22T08:38:56.175585+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:21.659229+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:21.659229+0000","last_peered":"2024-04-22T08:38:21.659229+0000","last_clean":"2024-04-22T08:38:21.659229+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:21.659229+0000","last_undegraded":"2024-04-22T08:38:21.659229+0000","last_fullsized":"2024-04-22T08:38:21.659229+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20874,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.668994+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:38:56.668994+0000","last_peered":"2024-04-22T08:38:56.668994+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:38:56.668994+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569282+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:18.569282+0000","last_peered":"2024-04-22T08:38:18.569282+0000","last_clean":"2024-04-22T08:38:18.569282+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:18.569282+0000","last_undegraded":"2024-04-22T08:38:18.569282+0000","last_fullsized":"2024-04-22T08:38:18.569282+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27604,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569476+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:18.569476+0000","last_peered":"2024-04-22T08:38:18.569476+0000","last_clean":"2024-04-22T08:38:18.569476+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:18.569476+0000","last_undegraded":"2024-04-22T08:38:18.569476+0000","last_fullsized":"2024-04-22T08:38:18.569476+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27623,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.568128+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:18.568128+0000","last_peered":"2024-04-22T08:38:18.568128+0000","last_clean":"2024-04-22T08:38:18.568128+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:18.568128+0000","last_undegraded":"2024-04-22T08:38:18.568128+0000","last_fullsized":"2024-04-22T08:38:18.568128+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-22T08:38:18.569079+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:18.569079+0000","last_peered":"2024-04-22T08:38:18.569079+0000","last_clean":"2024-04-22T08:38:18.569079+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:18.569079+0000","last_undegraded":"2024-04-22T08:38:18.569079+0000","last_fullsized":"2024-04-22T08:38:18.569079+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28154,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117139,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478364,"kb_used_data":121872,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893476,"statfs":{"total":96636764160,"available":96146919424,"internally_reserved":0,"allocated":124796928,"data_stored":124527456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.34499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509488,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862352,"statfs":{"total":96636764160,"available":96115048448,"internally_reserved":0,"allocated":430080,"data_stored":139734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313368,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:04.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:05.058 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:05.059 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:05.078 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-04-22T08:39:04.463116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491620,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623900,"statfs":{"total":289910292480,"available":288382873600,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.289431"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":22477,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:01.673231+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:01.673231+0000","last_peered":"2024-04-22T08:39:01.673231+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:01.673231+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27624,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174654+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:56.174654+0000","last_peered":"2024-04-22T08:38:56.174654+0000","last_clean":"2024-04-22T08:38:56.174654+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:56.174654+0000","last_undegraded":"2024-04-22T08:38:56.174654+0000","last_fullsized":"2024-04-22T08:38:56.174654+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.314},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.5}]}]},{"osd":1,"up_from":33,"seq":141733920902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501300,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870540,"statfs":{"total":96636764160,"available":96123432960,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.502},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313369,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:05.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:05.407 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:05.408 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:05.432 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-04-22T08:39:04.463116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491620,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623900,"statfs":{"total":289910292480,"available":288382873600,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.289431"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":22477,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:01.673231+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:01.673231+0000","last_peered":"2024-04-22T08:39:01.673231+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:01.673231+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27624,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174654+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:56.174654+0000","last_peered":"2024-04-22T08:38:56.174654+0000","last_clean":"2024-04-22T08:38:56.174654+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:56.174654+0000","last_undegraded":"2024-04-22T08:38:56.174654+0000","last_fullsized":"2024-04-22T08:38:56.174654+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.314},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.5}]}]},{"osd":1,"up_from":33,"seq":141733920902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501300,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870540,"statfs":{"total":96636764160,"available":96123432960,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.502},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313369,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:05.433 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:05.433 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:05.433 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:05.433 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 22477, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:01.673231+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:01.673231+0000', 'last_peered': '2024-04-22T08:39:01.673231+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:01.673231+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2832, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1063, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:05.433 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:05.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:05.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:05.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:05.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:05.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:05.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:05.804 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:05.804 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16652/26 objects degraded (64046.154%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":780,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527418880,"bytes_avail":288382873600,"bytes_total":289910292480,"degraded_objects":16652,"degraded_total":26,"degraded_ratio":640.46153846153845,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:05.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:05.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:06.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:06.168 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-04-22T08:39:04.463116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491620,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623900,"statfs":{"total":289910292480,"available":288382873600,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1072,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.289431"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":22477,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:01.673231+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:01.673231+0000","last_peered":"2024-04-22T08:39:01.673231+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:01.673231+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27624,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174654+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:56.174654+0000","last_peered":"2024-04-22T08:38:56.174654+0000","last_clean":"2024-04-22T08:38:56.174654+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:56.174654+0000","last_undegraded":"2024-04-22T08:38:56.174654+0000","last_fullsized":"2024-04-22T08:38:56.174654+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28955,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.314},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.5}]}]},{"osd":1,"up_from":33,"seq":141733920902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501300,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870540,"statfs":{"total":96636764160,"available":96123432960,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.502},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.60199999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313369,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:09.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:09.540 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:09.557 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-04-22T08:39:08.464645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1363,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.279241"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":23243,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:06.669428+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:06.669428+0000","last_peered":"2024-04-22T08:39:06.669428+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:06.669428+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117141,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.371},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.45600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.72099999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313370,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:09.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:09.877 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:09.895 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-04-22T08:39:08.464645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1363,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.279241"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":23243,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:06.669428+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:06.669428+0000","last_peered":"2024-04-22T08:39:06.669428+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:06.669428+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117141,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.371},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.45600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.72099999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313370,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 23243, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:06.669428+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:06.669428+0000', 'last_peered': '2024-04-22T08:39:06.669428+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:06.669428+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2449, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1446, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:09.897 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:09.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:09.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:09.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:09.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:10.256 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16269/26 objects degraded (62573.077%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531613184,"bytes_avail":288378679296,"bytes_total":289910292480,"degraded_objects":16269,"degraded_total":26,"degraded_ratio":625.73076923076928,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1363,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:10.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:10.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:10.566 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:10.582 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-04-22T08:39:10.465331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003827"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":23243,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:06.669428+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:06.669428+0000","last_peered":"2024-04-22T08:39:06.669428+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:06.669428+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29338,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117141,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.371},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.01,"15min":1.1060000000000001},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68200000000000005,"5min":15.6,"15min":15.6},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.109,"15min":1.038},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20499999999999999},"max":{"1min":6.6239999999999997,"5min":7.556,"15min":18.32},"last":0.45600000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.72099999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313370,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:13.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:13.922 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:13.923 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:13.940 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-04-22T08:39:12.465782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24007,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:11.676780+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:11.676780+0000","last_peered":"2024-04-22T08:39:11.676780+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:11.676780+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1828,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2065}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.72199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.68000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.72099999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313370,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":1.3049999999999999,"15min":0.97399999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.16},"max":{"1min":0.64000000000000001,"5min":11.368,"15min":11.368},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.64700000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:13.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:15.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:15.087 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:15.109 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-22T08:39:14.466564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24007,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:11.676780+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:11.676780+0000","last_peered":"2024-04-22T08:39:11.676780+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:11.676780+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1828,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2065}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.72199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.68000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.222},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.26200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313371,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 24007, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:11.676780+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:11.676780+0000', 'last_peered': '2024-04-22T08:39:11.676780+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:11.676780+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2067, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1828, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2065}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:15.111 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:15.112 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:15.112 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:15.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:15.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:15.495 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15887/26 objects degraded (61103.846%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1531613184,"bytes_avail":288378679296,"bytes_total":289910292480,"degraded_objects":15887,"degraded_total":26,"degraded_ratio":611.03846153846155,"recovering_objects_per_sec":63,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":765,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:15.514 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:15.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:15.860 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:15.861 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:15.880 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-22T08:39:14.466564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495716,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281619804,"statfs":{"total":289910292480,"available":288378679296,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24007,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:11.676780+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:11.676780+0000","last_peered":"2024-04-22T08:39:11.676780+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:11.676780+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1828,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2065}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29720,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.72199999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.68000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505396,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866444,"statfs":{"total":96636764160,"available":96119238656,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.84399999999999997,"15min":1.2809999999999999},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.60599999999999998,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.222},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.91400000000000003,"15min":1.1859999999999999},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.70499999999999996,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.26200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313371,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:18.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:19.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:19.230 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:19.247 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-04-22T08:39:18.468048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499812,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615708,"statfs":{"total":289910292480,"available":288374484992,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004072"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24829,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:16.682549+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:16.682549+0000","last_peered":"2024-04-22T08:39:16.682549+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:16.682549+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1654}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117143,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.79600000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.749}]}]},{"osd":1,"up_from":33,"seq":141733920905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509492,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862348,"statfs":{"total":96636764160,"available":96115044352,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313372,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:19.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:19.578 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:19.578 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:19.601 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-04-22T08:39:18.468048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499812,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615708,"statfs":{"total":289910292480,"available":288374484992,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004072"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24829,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:16.682549+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:16.682549+0000","last_peered":"2024-04-22T08:39:16.682549+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:16.682549+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1654}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117143,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.79600000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.749}]}]},{"osd":1,"up_from":33,"seq":141733920905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509492,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862348,"statfs":{"total":96636764160,"available":96115044352,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313372,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:19.602 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:19.602 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:19.602 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:19.602 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 24829, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:16.682549+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:16.682549+0000', 'last_peered': '2024-04-22T08:39:16.682549+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:16.682549+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1656, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2239, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:19.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:19.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:19.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:19.950 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15476/26 objects degraded (59523.077%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535807488,"bytes_avail":288374484992,"bytes_total":289910292480,"degraded_objects":15476,"degraded_total":26,"degraded_ratio":595.23076923076928,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1176,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:19.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:19.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:20.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:20.270 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:20.288 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-04-22T08:39:18.468048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499812,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615708,"statfs":{"total":289910292480,"available":288374484992,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004072"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":24829,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:16.682549+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:16.682549+0000","last_peered":"2024-04-22T08:39:16.682549+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:16.682549+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1654}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30131,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117143,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.79600000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.749}]}]},{"osd":1,"up_from":33,"seq":141733920905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509492,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862348,"statfs":{"total":96636764160,"available":96115044352,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313372,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:23.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:23.613 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:23.614 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:23.632 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-04-22T08:39:22.469598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499812,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615708,"statfs":{"total":289910292480,"available":288374484992,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004267"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":25749,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:21.678014+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:21.678014+0000","last_peered":"2024-04-22T08:39:21.678014+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:21.678014+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2699,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.67000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509492,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862348,"statfs":{"total":96636764160,"available":96115044352,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313372,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:23.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:23.947 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:23.948 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-04-22T08:39:22.469598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499812,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615708,"statfs":{"total":289910292480,"available":288374484992,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004267"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":25749,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:21.678014+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:21.678014+0000","last_peered":"2024-04-22T08:39:21.678014+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:21.678014+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2699,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21079,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.176388+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:56.176388+0000","last_peered":"2024-04-22T08:38:56.176388+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:56.176388+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175854+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:56.175854+0000","last_peered":"2024-04-22T08:38:56.175854+0000","last_clean":"2024-04-22T08:38:56.175854+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:56.175854+0000","last_undegraded":"2024-04-22T08:38:56.175854+0000","last_fullsized":"2024-04-22T08:38:56.175854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.67000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509492,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862348,"statfs":{"total":96636764160,"available":96115044352,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.008,"15min":0.76500000000000001},"min":{"1min":0.19500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.86599999999999999,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.498},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.96899999999999997,"15min":0.94099999999999995},"min":{"1min":0.22800000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.95099999999999996,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313372,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 25749, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:21.678014+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:21.678014+0000', 'last_peered': '2024-04-22T08:39:21.678014+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:21.678014+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2699, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21079, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.176388+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:56.176388+0000', 'last_peered': '2024-04-22T08:38:56.176388+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:56.176388+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:23.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:23.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:24.345 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15016/26 objects degraded (57753.846%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":799,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535807488,"bytes_avail":288374484992,"bytes_total":289910292480,"degraded_objects":15016,"degraded_total":26,"degraded_ratio":577.53846153846155,"recovering_objects_per_sec":104,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:24.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:24.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:24.672 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:24.689 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-04-22T08:39:24.470371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487524,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281627996,"statfs":{"total":289910292480,"available":288387067904,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004588"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":25749,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:21.678014+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:21.678014+0000","last_peered":"2024-04-22T08:39:21.678014+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:21.678014+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2699,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.67000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497204,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874636,"statfs":{"total":96636764160,"available":96127627264,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313373,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.375},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:27.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:28.001 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:28.002 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:28.019 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-04-22T08:39:26.471087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487524,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281627996,"statfs":{"total":289910292480,"available":288387067904,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004522"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":25749,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:21.678014+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:21.678014+0000","last_peered":"2024-04-22T08:39:21.678014+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:21.678014+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2699,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.67000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497204,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874636,"statfs":{"total":96636764160,"available":96127627264,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313373,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.375},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:28.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:28.313 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:28.314 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:28.332 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-04-22T08:39:26.471087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487524,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281627996,"statfs":{"total":289910292480,"available":288387067904,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004522"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27998,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178069+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:56.178069+0000","last_peered":"2024-04-22T08:38:56.178069+0000","last_clean":"2024-04-22T08:38:56.178069+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:56.178069+0000","last_undegraded":"2024-04-22T08:38:56.178069+0000","last_fullsized":"2024-04-22T08:38:56.178069+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":25749,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:21.678014+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:21.678014+0000","last_peered":"2024-04-22T08:39:21.678014+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:21.678014+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2699,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30591,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":0.79200000000000004,"5min":6.3860000000000001,"15min":13.635},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.89400000000000002,"5min":6.431,"15min":13.721},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.67000000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497204,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874636,"statfs":{"total":96636764160,"available":96127627264,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.59299999999999997}]}]},{"osd":0,"up_from":21,"seq":90194313373,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.375},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 25749, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:21.678014+0000', 'last_change': '2024-04-22T08:38:55.971999+0000', 'last_active': '2024-04-22T08:39:21.678014+0000', 'last_peered': '2024-04-22T08:39:21.678014+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:21.678014+0000', 'last_undegraded': '2024-04-22T08:27:56.202476+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 34, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2699, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21080, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:57.182509+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:57.182509+0000', 'last_peered': '2024-04-22T08:38:57.182509+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:57.182509+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:28.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:28.335 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:28.335 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:28.335 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:28.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:28.705 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:28.705 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14479/26 objects degraded (55688.462%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527418880,"bytes_avail":288382873600,"bytes_total":289910292480,"degraded_objects":14479,"degraded_total":26,"degraded_ratio":556.88461538461536,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1408,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:28.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:28.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:29.060 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-04-22T08:39:28.471689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31128,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491620,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623900,"statfs":{"total":289910292480,"available":288382873600,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004299"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27999,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185794+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:57.185794+0000","last_peered":"2024-04-22T08:38:57.185794+0000","last_clean":"2024-04-22T08:38:57.185794+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:57.185794+0000","last_undegraded":"2024-04-22T08:38:57.185794+0000","last_fullsized":"2024-04-22T08:38:57.185794+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":26823,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:26.684674+0000","last_change":"2024-04-22T08:38:55.971999+0000","last_active":"2024-04-22T08:39:26.684674+0000","last_peered":"2024-04-22T08:39:26.684674+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:26.684674+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31128,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186671104,"data_stored":186639793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117145,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.499},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.56000000000000005}]}]},{"osd":1,"up_from":33,"seq":141733920907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501300,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870540,"statfs":{"total":96636764160,"available":96123432960,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313374,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":124395520,"data_stored":124392041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:32.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:32.447 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:32.448 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:32.472 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-04-22T08:39:31.730495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1491620,"kb_used_data":183488,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281623900,"statfs":{"total":289910292480,"available":288382873600,"internally_reserved":0,"allocated":187891712,"data_stored":187052810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1656,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.261499"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20415,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178434+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:56.178434+0000","last_peered":"2024-04-22T08:38:56.178434+0000","last_clean":"2024-04-22T08:38:56.178434+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:56.178434+0000","last_undegraded":"2024-04-22T08:38:56.178434+0000","last_fullsized":"2024-04-22T08:38:56.178434+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":27999,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185794+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:38:57.185794+0000","last_peered":"2024-04-22T08:38:57.185794+0000","last_clean":"2024-04-22T08:38:57.185794+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:38:57.185794+0000","last_undegraded":"2024-04-22T08:38:57.185794+0000","last_fullsized":"2024-04-22T08:38:57.185794+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27616,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185208+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.185208+0000","last_peered":"2024-04-22T08:38:57.185208+0000","last_clean":"2024-04-22T08:38:57.185208+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.185208+0000","last_undegraded":"2024-04-22T08:38:57.185208+0000","last_fullsized":"2024-04-22T08:38:57.185208+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177796+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:56.177796+0000","last_peered":"2024-04-22T08:38:56.177796+0000","last_clean":"2024-04-22T08:38:56.177796+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:56.177796+0000","last_undegraded":"2024-04-22T08:38:56.177796+0000","last_fullsized":"2024-04-22T08:38:56.177796+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":28142,"reported_epoch":49,"state":"active+undersized+remapped","last_fresh":"2024-04-22T08:39:31.309670+0000","last_change":"2024-04-22T08:39:31.310068+0000","last_active":"2024-04-22T08:39:31.309670+0000","last_peered":"2024-04-22T08:39:31.309670+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.309670+0000","last_undegraded":"2024-04-22T08:27:56.202476+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":34,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.177786+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:56.177786+0000","last_peered":"2024-04-22T08:38:56.177786+0000","last_clean":"2024-04-22T08:38:56.177786+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:56.177786+0000","last_undegraded":"2024-04-22T08:38:56.177786+0000","last_fullsized":"2024-04-22T08:38:56.177786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178372+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:56.178372+0000","last_peered":"2024-04-22T08:38:56.178372+0000","last_clean":"2024-04-22T08:38:56.178372+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:56.178372+0000","last_undegraded":"2024-04-22T08:38:56.178372+0000","last_fullsized":"2024-04-22T08:38:56.178372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20657,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178517+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:56.178517+0000","last_peered":"2024-04-22T08:38:56.178517+0000","last_clean":"2024-04-22T08:38:56.178517+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:56.178517+0000","last_undegraded":"2024-04-22T08:38:56.178517+0000","last_fullsized":"2024-04-22T08:38:56.178517+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178614+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:56.178614+0000","last_peered":"2024-04-22T08:38:56.178614+0000","last_clean":"2024-04-22T08:38:56.178614+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:56.178614+0000","last_undegraded":"2024-04-22T08:38:56.178614+0000","last_fullsized":"2024-04-22T08:38:56.178614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":71,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.178762+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:56.178762+0000","last_peered":"2024-04-22T08:38:56.178762+0000","last_clean":"2024-04-22T08:38:56.178762+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:56.178762+0000","last_undegraded":"2024-04-22T08:38:56.178762+0000","last_fullsized":"2024-04-22T08:38:56.178762+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20731,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.299234+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.299234+0000","last_peered":"2024-04-22T08:38:57.299234+0000","last_clean":"2024-04-22T08:38:57.299234+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.299234+0000","last_undegraded":"2024-04-22T08:38:57.299234+0000","last_fullsized":"2024-04-22T08:38:57.299234+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186195968,"data_stored":186164657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501300,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870540,"statfs":{"total":96636764160,"available":96123432960,"internally_reserved":0,"allocated":438272,"data_stored":142568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313374,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:32.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:32.810 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:32.810 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-04-22T08:39:32.736200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499816,"kb_used_data":183492,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615704,"statfs":{"total":289910292480,"available":288374480896,"internally_reserved":0,"allocated":187895808,"data_stored":187055023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.265829"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20416,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185307+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:57.185307+0000","last_peered":"2024-04-22T08:38:57.185307+0000","last_clean":"2024-04-22T08:38:57.185307+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:57.185307+0000","last_undegraded":"2024-04-22T08:38:57.185307+0000","last_fullsized":"2024-04-22T08:38:57.185307+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28000,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:39:27.553282+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:27.553282+0000","last_peered":"2024-04-22T08:39:27.553282+0000","last_clean":"2024-04-22T08:39:27.553282+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:27.553282+0000","last_undegraded":"2024-04-22T08:39:27.553282+0000","last_fullsized":"2024-04-22T08:39:27.553282+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27617,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543602+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.543602+0000","last_peered":"2024-04-22T08:38:57.543602+0000","last_clean":"2024-04-22T08:38:57.543602+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.543602+0000","last_undegraded":"2024-04-22T08:38:57.543602+0000","last_fullsized":"2024-04-22T08:38:57.543602+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185039+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:57.185039+0000","last_peered":"2024-04-22T08:38:57.185039+0000","last_clean":"2024-04-22T08:38:57.185039+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:57.185039+0000","last_undegraded":"2024-04-22T08:38:57.185039+0000","last_fullsized":"2024-04-22T08:38:57.185039+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":50,"state":"peering","last_fresh":"2024-04-22T08:39:31.310068+0000","last_change":"2024-04-22T08:39:32.553938+0000","last_active":"2024-04-22T08:39:32.553938+0000","last_peered":"2024-04-22T08:39:31.310068+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.310068+0000","last_undegraded":"2024-04-22T08:39:31.310068+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185232+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:57.185232+0000","last_peered":"2024-04-22T08:38:57.185232+0000","last_clean":"2024-04-22T08:38:57.185232+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:57.185232+0000","last_undegraded":"2024-04-22T08:38:57.185232+0000","last_fullsized":"2024-04-22T08:38:57.185232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185498+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:57.185498+0000","last_peered":"2024-04-22T08:38:57.185498+0000","last_clean":"2024-04-22T08:38:57.185498+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:57.185498+0000","last_undegraded":"2024-04-22T08:38:57.185498+0000","last_fullsized":"2024-04-22T08:38:57.185498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20658,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185385+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:57.185385+0000","last_peered":"2024-04-22T08:38:57.185385+0000","last_clean":"2024-04-22T08:38:57.185385+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:57.185385+0000","last_undegraded":"2024-04-22T08:38:57.185385+0000","last_fullsized":"2024-04-22T08:38:57.185385+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185478+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:57.185478+0000","last_peered":"2024-04-22T08:38:57.185478+0000","last_clean":"2024-04-22T08:38:57.185478+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:57.185478+0000","last_undegraded":"2024-04-22T08:38:57.185478+0000","last_fullsized":"2024-04-22T08:38:57.185478+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185602+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:57.185602+0000","last_peered":"2024-04-22T08:38:57.185602+0000","last_clean":"2024-04-22T08:38:57.185602+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:57.185602+0000","last_undegraded":"2024-04-22T08:38:57.185602+0000","last_fullsized":"2024-04-22T08:38:57.185602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20732,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543684+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.543684+0000","last_peered":"2024-04-22T08:38:57.543684+0000","last_clean":"2024-04-22T08:38:57.543684+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.543684+0000","last_undegraded":"2024-04-22T08:38:57.543684+0000","last_fullsized":"2024-04-22T08:38:57.543684+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186191872,"data_stored":186164040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509496,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862344,"statfs":{"total":96636764160,"available":96115040256,"internally_reserved":0,"allocated":442368,"data_stored":144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56299999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313374,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20899, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180883+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:38:56.180883+0000', 'last_peered': '2024-04-22T08:38:56.180883+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:38:56.180883+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20685, 'reported_epoch': 50, 'state': 'peering', 'last_fresh': '2024-04-22T08:39:31.310068+0000', 'last_change': '2024-04-22T08:39:32.553938+0000', 'last_active': '2024-04-22T08:39:32.553938+0000', 'last_peered': '2024-04-22T08:39:31.310068+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:31.310068+0000', 'last_undegraded': '2024-04-22T08:39:31.310068+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 50, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3895, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21080, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:57.182509+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:38:57.182509+0000', 'last_peered': '2024-04-22T08:38:57.182509+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:38:57.182509+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 20909, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.180103+0000', 'last_change': '2024-04-22T08:27:56.688356+0000', 'last_active': '2024-04-22T08:38:56.180103+0000', 'last_peered': '2024-04-22T08:38:56.180103+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:38:56.180103+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 338, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:32.837 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20435, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:38:56.175926+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:38:56.175926+0000', 'last_peered': '2024-04-22T08:38:56.175926+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:38:56.175926+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:32.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:33.246 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:33.246 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13820/26 objects degraded (53153.846%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1527418880,"bytes_avail":288382873600,"bytes_total":289910292480,"degraded_objects":13820,"degraded_total":26,"degraded_ratio":531.53846153846155,"recovering_objects_per_sec":119,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1196,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:33.269 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:33.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:33.607 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:33.628 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-04-22T08:39:32.736200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499816,"kb_used_data":183492,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615704,"statfs":{"total":289910292480,"available":288374480896,"internally_reserved":0,"allocated":187895808,"data_stored":187055023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.265829"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20899,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180883+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:38:56.180883+0000","last_peered":"2024-04-22T08:38:56.180883+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:38:56.180883+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20416,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185307+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:57.185307+0000","last_peered":"2024-04-22T08:38:57.185307+0000","last_clean":"2024-04-22T08:38:57.185307+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:57.185307+0000","last_undegraded":"2024-04-22T08:38:57.185307+0000","last_fullsized":"2024-04-22T08:38:57.185307+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20073,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175531+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:38:56.175531+0000","last_peered":"2024-04-22T08:38:56.175531+0000","last_clean":"2024-04-22T08:38:56.175531+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:38:56.175531+0000","last_undegraded":"2024-04-22T08:38:56.175531+0000","last_fullsized":"2024-04-22T08:38:56.175531+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28000,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:39:27.553282+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:27.553282+0000","last_peered":"2024-04-22T08:39:27.553282+0000","last_clean":"2024-04-22T08:39:27.553282+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:27.553282+0000","last_undegraded":"2024-04-22T08:39:27.553282+0000","last_fullsized":"2024-04-22T08:39:27.553282+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27617,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543602+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.543602+0000","last_peered":"2024-04-22T08:38:57.543602+0000","last_clean":"2024-04-22T08:38:57.543602+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.543602+0000","last_undegraded":"2024-04-22T08:38:57.543602+0000","last_fullsized":"2024-04-22T08:38:57.543602+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19966,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175090+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:38:56.175090+0000","last_peered":"2024-04-22T08:38:56.175090+0000","last_clean":"2024-04-22T08:38:56.175090+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:38:56.175090+0000","last_undegraded":"2024-04-22T08:38:56.175090+0000","last_fullsized":"2024-04-22T08:38:56.175090+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185039+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:57.185039+0000","last_peered":"2024-04-22T08:38:57.185039+0000","last_clean":"2024-04-22T08:38:57.185039+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:57.185039+0000","last_undegraded":"2024-04-22T08:38:57.185039+0000","last_fullsized":"2024-04-22T08:38:57.185039+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":50,"state":"peering","last_fresh":"2024-04-22T08:39:31.310068+0000","last_change":"2024-04-22T08:39:32.553938+0000","last_active":"2024-04-22T08:39:32.553938+0000","last_peered":"2024-04-22T08:39:31.310068+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.310068+0000","last_undegraded":"2024-04-22T08:39:31.310068+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21080,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:57.182509+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:38:57.182509+0000","last_peered":"2024-04-22T08:38:57.182509+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:38:57.182509+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":84,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.182034+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:38:57.182034+0000","last_peered":"2024-04-22T08:38:57.182034+0000","last_clean":"2024-04-22T08:38:57.182034+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:38:57.182034+0000","last_undegraded":"2024-04-22T08:38:57.182034+0000","last_fullsized":"2024-04-22T08:38:57.182034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185232+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:57.185232+0000","last_peered":"2024-04-22T08:38:57.185232+0000","last_clean":"2024-04-22T08:38:57.185232+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:57.185232+0000","last_undegraded":"2024-04-22T08:38:57.185232+0000","last_fullsized":"2024-04-22T08:38:57.185232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20909,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.180103+0000","last_change":"2024-04-22T08:27:56.688356+0000","last_active":"2024-04-22T08:38:56.180103+0000","last_peered":"2024-04-22T08:38:56.180103+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:38:56.180103+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185498+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:57.185498+0000","last_peered":"2024-04-22T08:38:57.185498+0000","last_clean":"2024-04-22T08:38:57.185498+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:57.185498+0000","last_undegraded":"2024-04-22T08:38:57.185498+0000","last_fullsized":"2024-04-22T08:38:57.185498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20435,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:38:56.175926+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:38:56.175926+0000","last_peered":"2024-04-22T08:38:56.175926+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:38:56.175926+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20658,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185385+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:57.185385+0000","last_peered":"2024-04-22T08:38:57.185385+0000","last_clean":"2024-04-22T08:38:57.185385+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:57.185385+0000","last_undegraded":"2024-04-22T08:38:57.185385+0000","last_fullsized":"2024-04-22T08:38:57.185385+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185478+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:57.185478+0000","last_peered":"2024-04-22T08:38:57.185478+0000","last_clean":"2024-04-22T08:38:57.185478+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:57.185478+0000","last_undegraded":"2024-04-22T08:38:57.185478+0000","last_fullsized":"2024-04-22T08:38:57.185478+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185602+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:57.185602+0000","last_peered":"2024-04-22T08:38:57.185602+0000","last_clean":"2024-04-22T08:38:57.185602+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:57.185602+0000","last_undegraded":"2024-04-22T08:38:57.185602+0000","last_fullsized":"2024-04-22T08:38:57.185602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20732,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543684+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.543684+0000","last_peered":"2024-04-22T08:38:57.543684+0000","last_clean":"2024-04-22T08:38:57.543684+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.543684+0000","last_undegraded":"2024-04-22T08:38:57.543684+0000","last_fullsized":"2024-04-22T08:38:57.543684+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31787,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186191872,"data_stored":186164040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509496,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862344,"statfs":{"total":96636764160,"available":96115040256,"internally_reserved":0,"allocated":442368,"data_stored":144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56299999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313374,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511948,"kb_used_data":61180,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859892,"statfs":{"total":96636764160,"available":96112529408,"internally_reserved":0,"allocated":62648320,"data_stored":62379952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.252,"5min":0.216,"15min":0.18099999999999999},"max":{"1min":0.78600000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.313},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.4550000000000001,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.75900000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:36.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:37.016 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-04-22T08:39:36.473855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499820,"kb_used_data":183500,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615700,"statfs":{"total":289910292480,"available":288374476800,"internally_reserved":0,"allocated":187904000,"data_stored":187057853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002165"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20416,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185307+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:57.185307+0000","last_peered":"2024-04-22T08:38:57.185307+0000","last_clean":"2024-04-22T08:38:57.185307+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:57.185307+0000","last_undegraded":"2024-04-22T08:38:57.185307+0000","last_fullsized":"2024-04-22T08:38:57.185307+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28000,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:39:27.553282+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:27.553282+0000","last_peered":"2024-04-22T08:39:27.553282+0000","last_clean":"2024-04-22T08:39:27.553282+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:27.553282+0000","last_undegraded":"2024-04-22T08:39:27.553282+0000","last_fullsized":"2024-04-22T08:39:27.553282+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27617,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543602+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.543602+0000","last_peered":"2024-04-22T08:38:57.543602+0000","last_clean":"2024-04-22T08:38:57.543602+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.543602+0000","last_undegraded":"2024-04-22T08:38:57.543602+0000","last_fullsized":"2024-04-22T08:38:57.543602+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185039+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:57.185039+0000","last_peered":"2024-04-22T08:38:57.185039+0000","last_clean":"2024-04-22T08:38:57.185039+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:57.185039+0000","last_undegraded":"2024-04-22T08:38:57.185039+0000","last_fullsized":"2024-04-22T08:38:57.185039+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":50,"state":"peering","last_fresh":"2024-04-22T08:39:31.310068+0000","last_change":"2024-04-22T08:39:32.553938+0000","last_active":"2024-04-22T08:39:32.553938+0000","last_peered":"2024-04-22T08:39:31.310068+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.310068+0000","last_undegraded":"2024-04-22T08:39:31.310068+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185232+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:57.185232+0000","last_peered":"2024-04-22T08:38:57.185232+0000","last_clean":"2024-04-22T08:38:57.185232+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:57.185232+0000","last_undegraded":"2024-04-22T08:38:57.185232+0000","last_fullsized":"2024-04-22T08:38:57.185232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":21270,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:32.761578+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:32.761578+0000","last_peered":"2024-04-22T08:39:32.761578+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:32.761578+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185498+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:57.185498+0000","last_peered":"2024-04-22T08:38:57.185498+0000","last_clean":"2024-04-22T08:38:57.185498+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:57.185498+0000","last_undegraded":"2024-04-22T08:38:57.185498+0000","last_fullsized":"2024-04-22T08:38:57.185498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20658,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185385+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:57.185385+0000","last_peered":"2024-04-22T08:38:57.185385+0000","last_clean":"2024-04-22T08:38:57.185385+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:57.185385+0000","last_undegraded":"2024-04-22T08:38:57.185385+0000","last_fullsized":"2024-04-22T08:38:57.185385+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185478+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:57.185478+0000","last_peered":"2024-04-22T08:38:57.185478+0000","last_clean":"2024-04-22T08:38:57.185478+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:57.185478+0000","last_undegraded":"2024-04-22T08:38:57.185478+0000","last_fullsized":"2024-04-22T08:38:57.185478+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185602+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:57.185602+0000","last_peered":"2024-04-22T08:38:57.185602+0000","last_clean":"2024-04-22T08:38:57.185602+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:57.185602+0000","last_undegraded":"2024-04-22T08:38:57.185602+0000","last_fullsized":"2024-04-22T08:38:57.185602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20732,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543684+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.543684+0000","last_peered":"2024-04-22T08:38:57.543684+0000","last_clean":"2024-04-22T08:38:57.543684+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.543684+0000","last_undegraded":"2024-04-22T08:38:57.543684+0000","last_fullsized":"2024-04-22T08:38:57.543684+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186191872,"data_stored":186164040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509496,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862344,"statfs":{"total":96636764160,"available":96115040256,"internally_reserved":0,"allocated":442368,"data_stored":144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56299999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313375,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511952,"kb_used_data":61188,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859888,"statfs":{"total":96636764160,"available":96112525312,"internally_reserved":0,"allocated":62656512,"data_stored":62382782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:37.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:37.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:37.402 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:37.421 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-04-22T08:39:36.473855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499820,"kb_used_data":183500,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615700,"statfs":{"total":289910292480,"available":288374476800,"internally_reserved":0,"allocated":187904000,"data_stored":187057853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002165"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20416,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185307+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:57.185307+0000","last_peered":"2024-04-22T08:38:57.185307+0000","last_clean":"2024-04-22T08:38:57.185307+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:57.185307+0000","last_undegraded":"2024-04-22T08:38:57.185307+0000","last_fullsized":"2024-04-22T08:38:57.185307+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28000,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:39:27.553282+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:27.553282+0000","last_peered":"2024-04-22T08:39:27.553282+0000","last_clean":"2024-04-22T08:39:27.553282+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:27.553282+0000","last_undegraded":"2024-04-22T08:39:27.553282+0000","last_fullsized":"2024-04-22T08:39:27.553282+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27617,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543602+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.543602+0000","last_peered":"2024-04-22T08:38:57.543602+0000","last_clean":"2024-04-22T08:38:57.543602+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.543602+0000","last_undegraded":"2024-04-22T08:38:57.543602+0000","last_fullsized":"2024-04-22T08:38:57.543602+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185039+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:57.185039+0000","last_peered":"2024-04-22T08:38:57.185039+0000","last_clean":"2024-04-22T08:38:57.185039+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:57.185039+0000","last_undegraded":"2024-04-22T08:38:57.185039+0000","last_fullsized":"2024-04-22T08:38:57.185039+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":50,"state":"peering","last_fresh":"2024-04-22T08:39:31.310068+0000","last_change":"2024-04-22T08:39:32.553938+0000","last_active":"2024-04-22T08:39:32.553938+0000","last_peered":"2024-04-22T08:39:31.310068+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.310068+0000","last_undegraded":"2024-04-22T08:39:31.310068+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185232+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:57.185232+0000","last_peered":"2024-04-22T08:38:57.185232+0000","last_clean":"2024-04-22T08:38:57.185232+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:57.185232+0000","last_undegraded":"2024-04-22T08:38:57.185232+0000","last_fullsized":"2024-04-22T08:38:57.185232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":21270,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:32.761578+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:32.761578+0000","last_peered":"2024-04-22T08:39:32.761578+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:32.761578+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185498+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:57.185498+0000","last_peered":"2024-04-22T08:38:57.185498+0000","last_clean":"2024-04-22T08:38:57.185498+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:57.185498+0000","last_undegraded":"2024-04-22T08:38:57.185498+0000","last_fullsized":"2024-04-22T08:38:57.185498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20658,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185385+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:57.185385+0000","last_peered":"2024-04-22T08:38:57.185385+0000","last_clean":"2024-04-22T08:38:57.185385+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:57.185385+0000","last_undegraded":"2024-04-22T08:38:57.185385+0000","last_fullsized":"2024-04-22T08:38:57.185385+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185478+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:57.185478+0000","last_peered":"2024-04-22T08:38:57.185478+0000","last_clean":"2024-04-22T08:38:57.185478+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:57.185478+0000","last_undegraded":"2024-04-22T08:38:57.185478+0000","last_fullsized":"2024-04-22T08:38:57.185478+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185602+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:57.185602+0000","last_peered":"2024-04-22T08:38:57.185602+0000","last_clean":"2024-04-22T08:38:57.185602+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:57.185602+0000","last_undegraded":"2024-04-22T08:38:57.185602+0000","last_fullsized":"2024-04-22T08:38:57.185602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20732,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543684+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.543684+0000","last_peered":"2024-04-22T08:38:57.543684+0000","last_clean":"2024-04-22T08:38:57.543684+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.543684+0000","last_undegraded":"2024-04-22T08:38:57.543684+0000","last_fullsized":"2024-04-22T08:38:57.543684+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186191872,"data_stored":186164040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509496,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862344,"statfs":{"total":96636764160,"available":96115040256,"internally_reserved":0,"allocated":442368,"data_stored":144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56299999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313375,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511952,"kb_used_data":61188,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859888,"statfs":{"total":96636764160,"available":96112525312,"internally_reserved":0,"allocated":62656512,"data_stored":62382782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:37.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:37.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20901, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735551+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:31.735551+0000', 'last_peered': '2024-04-22T08:39:31.735551+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:31.735551+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:37.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "41'20221", 'reported_seq': 20685, 'reported_epoch': 50, 'state': 'peering', 'last_fresh': '2024-04-22T08:39:31.310068+0000', 'last_change': '2024-04-22T08:39:32.553938+0000', 'last_active': '2024-04-22T08:39:32.553938+0000', 'last_peered': '2024-04-22T08:39:31.310068+0000', 'last_clean': '2024-04-22T08:27:54.070471+0000', 'last_became_active': '2024-04-22T08:27:56.654426+0000', 'last_became_peered': '2024-04-22T08:27:56.654426+0000', 'last_unstale': '2024-04-22T08:39:31.310068+0000', 'last_undegraded': '2024-04-22T08:39:31.310068+0000', 'last_fullsized': '2024-04-22T08:27:56.090398+0000', 'mapping_epoch': 50, 'log_start': "37'10200", 'ondisk_log_start': "37'10200", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 798, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20161, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3895, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21082, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.733929+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:31.733929+0000', 'last_peered': '2024-04-22T08:39:31.733929+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:31.733929+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 21270, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:32.761578+0000', 'last_change': '2024-04-22T08:39:31.310909+0000', 'last_active': '2024-04-22T08:39:32.761578+0000', 'last_peered': '2024-04-22T08:39:32.761578+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:39:32.761578+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 517, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:37.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20437, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735676+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:39:31.735676+0000', 'last_peered': '2024-04-22T08:39:31.735676+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:31.735676+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:37.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:37.892 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:37.892 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13641/26 objects degraded (52465.385%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1535815680,"bytes_avail":288374476800,"bytes_total":289910292480,"inactive_pgs_ratio":0.041666667908430099,"degraded_objects":13641,"degraded_total":26,"degraded_ratio":524.65384615384619,"recovering_objects_per_sec":104,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":838,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:37.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:37.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:38.257 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:38.257 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:38.276 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-04-22T08:39:36.473855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1499820,"kb_used_data":183500,"kb_used_omap":12,"kb_used_meta":1316275,"kb_avail":281615700,"statfs":{"total":289910292480,"available":288374476800,"internally_reserved":0,"allocated":187904000,"data_stored":187057853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1347865710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002165"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.669944+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:38:56.669944+0000","last_peered":"2024-04-22T08:38:56.669944+0000","last_clean":"2024-04-22T08:38:56.669944+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:38:56.669944+0000","last_undegraded":"2024-04-22T08:38:56.669944+0000","last_fullsized":"2024-04-22T08:38:56.669944+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20416,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185307+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:38:57.185307+0000","last_peered":"2024-04-22T08:38:57.185307+0000","last_clean":"2024-04-22T08:38:57.185307+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:38:57.185307+0000","last_undegraded":"2024-04-22T08:38:57.185307+0000","last_fullsized":"2024-04-22T08:38:57.185307+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175129+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:38:56.175129+0000","last_peered":"2024-04-22T08:38:56.175129+0000","last_clean":"2024-04-22T08:38:56.175129+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:38:56.175129+0000","last_undegraded":"2024-04-22T08:38:56.175129+0000","last_fullsized":"2024-04-22T08:38:56.175129+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28000,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:39:27.553282+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:27.553282+0000","last_peered":"2024-04-22T08:39:27.553282+0000","last_clean":"2024-04-22T08:39:27.553282+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:27.553282+0000","last_undegraded":"2024-04-22T08:39:27.553282+0000","last_fullsized":"2024-04-22T08:39:27.553282+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27617,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543602+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:38:57.543602+0000","last_peered":"2024-04-22T08:38:57.543602+0000","last_clean":"2024-04-22T08:38:57.543602+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:38:57.543602+0000","last_undegraded":"2024-04-22T08:38:57.543602+0000","last_fullsized":"2024-04-22T08:38:57.543602+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185039+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:38:57.185039+0000","last_peered":"2024-04-22T08:38:57.185039+0000","last_clean":"2024-04-22T08:38:57.185039+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:38:57.185039+0000","last_undegraded":"2024-04-22T08:38:57.185039+0000","last_fullsized":"2024-04-22T08:38:57.185039+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20685,"reported_epoch":50,"state":"peering","last_fresh":"2024-04-22T08:39:31.310068+0000","last_change":"2024-04-22T08:39:32.553938+0000","last_active":"2024-04-22T08:39:32.553938+0000","last_peered":"2024-04-22T08:39:31.310068+0000","last_clean":"2024-04-22T08:27:54.070471+0000","last_became_active":"2024-04-22T08:27:56.654426+0000","last_became_peered":"2024-04-22T08:27:56.654426+0000","last_unstale":"2024-04-22T08:39:31.310068+0000","last_undegraded":"2024-04-22T08:39:31.310068+0000","last_fullsized":"2024-04-22T08:27:56.090398+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.174949+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:38:56.174949+0000","last_peered":"2024-04-22T08:38:56.174949+0000","last_clean":"2024-04-22T08:38:56.174949+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:38:56.174949+0000","last_undegraded":"2024-04-22T08:38:56.174949+0000","last_fullsized":"2024-04-22T08:38:56.174949+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175027+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:38:56.175027+0000","last_peered":"2024-04-22T08:38:56.175027+0000","last_clean":"2024-04-22T08:38:56.175027+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:38:56.175027+0000","last_undegraded":"2024-04-22T08:38:56.175027+0000","last_fullsized":"2024-04-22T08:38:56.175027+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185232+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:38:57.185232+0000","last_peered":"2024-04-22T08:38:57.185232+0000","last_clean":"2024-04-22T08:38:57.185232+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:38:57.185232+0000","last_undegraded":"2024-04-22T08:38:57.185232+0000","last_fullsized":"2024-04-22T08:38:57.185232+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27625,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.191487+0000","last_change":"2024-04-22T08:32:19.167113+0000","last_active":"2024-04-22T08:38:57.191487+0000","last_peered":"2024-04-22T08:38:57.191487+0000","last_clean":"2024-04-22T08:38:57.191487+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:38:57.191487+0000","last_undegraded":"2024-04-22T08:38:57.191487+0000","last_fullsized":"2024-04-22T08:38:57.191487+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":21270,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:32.761578+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:32.761578+0000","last_peered":"2024-04-22T08:39:32.761578+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:32.761578+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185498+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:38:57.185498+0000","last_peered":"2024-04-22T08:38:57.185498+0000","last_clean":"2024-04-22T08:38:57.185498+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:38:57.185498+0000","last_undegraded":"2024-04-22T08:38:57.185498+0000","last_fullsized":"2024-04-22T08:38:57.185498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-22T08:38:56.175339+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:38:56.175339+0000","last_peered":"2024-04-22T08:38:56.175339+0000","last_clean":"2024-04-22T08:38:56.175339+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:38:56.175339+0000","last_undegraded":"2024-04-22T08:38:56.175339+0000","last_fullsized":"2024-04-22T08:38:56.175339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20658,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185385+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:38:57.185385+0000","last_peered":"2024-04-22T08:38:57.185385+0000","last_clean":"2024-04-22T08:38:57.185385+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:38:57.185385+0000","last_undegraded":"2024-04-22T08:38:57.185385+0000","last_fullsized":"2024-04-22T08:38:57.185385+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185478+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:38:57.185478+0000","last_peered":"2024-04-22T08:38:57.185478+0000","last_clean":"2024-04-22T08:38:57.185478+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:38:57.185478+0000","last_undegraded":"2024-04-22T08:38:57.185478+0000","last_fullsized":"2024-04-22T08:38:57.185478+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.185602+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:38:57.185602+0000","last_peered":"2024-04-22T08:38:57.185602+0000","last_clean":"2024-04-22T08:38:57.185602+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:38:57.185602+0000","last_undegraded":"2024-04-22T08:38:57.185602+0000","last_fullsized":"2024-04-22T08:38:57.185602+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20732,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-22T08:38:57.543684+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:38:57.543684+0000","last_peered":"2024-04-22T08:38:57.543684+0000","last_clean":"2024-04-22T08:38:57.543684+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:38:57.543684+0000","last_undegraded":"2024-04-22T08:38:57.543684+0000","last_fullsized":"2024-04-22T08:38:57.543684+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31966,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186191872,"data_stored":186164040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478372,"kb_used_data":121880,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93893468,"statfs":{"total":96636764160,"available":96146911232,"internally_reserved":0,"allocated":124805120,"data_stored":124530290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.626}]}]},{"osd":1,"up_from":33,"seq":141733920908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509496,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862344,"statfs":{"total":96636764160,"available":96115040256,"internally_reserved":0,"allocated":442368,"data_stored":144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":521268940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.56299999999999994}]}]},{"osd":0,"up_from":21,"seq":90194313375,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511952,"kb_used_data":61188,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859888,"statfs":{"total":96636764160,"available":96112525312,"internally_reserved":0,"allocated":62656512,"data_stored":62382782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123920384,"data_stored":123916905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:41.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:41.613 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:41.614 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:41.634 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-04-22T08:39:40.474606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1485184,"kb_used_data":181148,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281630336,"statfs":{"total":289910292480,"available":288389464064,"internally_reserved":0,"allocated":185495552,"data_stored":184627894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.744111"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27629,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:32.738766+0000","last_change":"2024-04-22T08:39:34.855329+0000","last_active":"2024-04-22T08:39:32.738766+0000","last_peered":"2024-04-22T08:39:32.738766+0000","last_clean":"2024-04-22T08:39:32.738766+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:32.738766+0000","last_undegraded":"2024-04-22T08:39:32.738766+0000","last_fullsized":"2024-04-22T08:39:32.738766+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":22790,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:37.762402+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:37.762402+0000","last_peered":"2024-04-22T08:39:37.762402+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:37.762402+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1277,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183734272,"data_stored":183706440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117147,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476004,"kb_used_data":119512,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93895836,"statfs":{"total":96636764160,"available":96149336064,"internally_reserved":0,"allocated":122380288,"data_stored":122094713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.311},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.45700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497216,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874624,"statfs":{"total":96636764160,"available":96127614976,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.435},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.47899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313376,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121462784,"data_stored":121459305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:41.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:41.998 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:41.999 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:42.020 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-04-22T08:39:40.474606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1485184,"kb_used_data":181148,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281630336,"statfs":{"total":289910292480,"available":288389464064,"internally_reserved":0,"allocated":185495552,"data_stored":184627894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.744111"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27629,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:32.738766+0000","last_change":"2024-04-22T08:39:34.855329+0000","last_active":"2024-04-22T08:39:32.738766+0000","last_peered":"2024-04-22T08:39:32.738766+0000","last_clean":"2024-04-22T08:39:32.738766+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:32.738766+0000","last_undegraded":"2024-04-22T08:39:32.738766+0000","last_fullsized":"2024-04-22T08:39:32.738766+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":22790,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:37.762402+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:37.762402+0000","last_peered":"2024-04-22T08:39:37.762402+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:37.762402+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1277,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183734272,"data_stored":183706440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117147,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476004,"kb_used_data":119512,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93895836,"statfs":{"total":96636764160,"available":96149336064,"internally_reserved":0,"allocated":122380288,"data_stored":122094713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.311},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.45700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497216,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874624,"statfs":{"total":96636764160,"available":96127614976,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.435},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.47899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313376,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121462784,"data_stored":121459305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:42.021 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:42.021 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20901, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735551+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:31.735551+0000', 'last_peered': '2024-04-22T08:39:31.735551+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:31.735551+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21082, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.733929+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:31.733929+0000', 'last_peered': '2024-04-22T08:39:31.733929+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:31.733929+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27629, 'reported_epoch': 51, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-22T08:39:32.738766+0000', 'last_change': '2024-04-22T08:39:34.855329+0000', 'last_active': '2024-04-22T08:39:32.738766+0000', 'last_peered': '2024-04-22T08:39:32.738766+0000', 'last_clean': '2024-04-22T08:39:32.738766+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:32.738766+0000', 'last_undegraded': '2024-04-22T08:39:32.738766+0000', 'last_fullsized': '2024-04-22T08:39:32.738766+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 22790, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:37.762402+0000', 'last_change': '2024-04-22T08:39:31.310909+0000', 'last_active': '2024-04-22T08:39:37.762402+0000', 'last_peered': '2024-04-22T08:39:37.762402+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:39:37.762402+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2447, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1277, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:42.022 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20437, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735676+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:39:31.735676+0000', 'last_peered': '2024-04-22T08:39:31.735676+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:31.735676+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:42.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:42.423 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12881/26 objects degraded (49542.308%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":817,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1520828416,"bytes_avail":288389464064,"bytes_total":289910292480,"degraded_objects":12881,"degraded_total":26,"degraded_ratio":495.42307692307691,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:42.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:42.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:42.762 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:42.763 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:42.782 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-04-22T08:39:42.474902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1482780,"kb_used_data":178748,"kb_used_omap":12,"kb_used_meta":1303987,"kb_avail":281632740,"statfs":{"total":289910292480,"available":288391925760,"internally_reserved":0,"allocated":183037952,"data_stored":182170294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1335282798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002001"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27630,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:34.855329+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:34.855329+0000","last_peered":"2024-04-22T08:39:34.855329+0000","last_clean":"2024-04-22T08:39:34.855329+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:34.855329+0000","last_undegraded":"2024-04-22T08:39:34.855329+0000","last_fullsized":"2024-04-22T08:39:34.855329+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":22790,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:37.762402+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:37.762402+0000","last_peered":"2024-04-22T08:39:37.762402+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:37.762402+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1277,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32726,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181276672,"data_stored":181248840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117148,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473600,"kb_used_data":117112,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93898240,"statfs":{"total":96636764160,"available":96151797760,"internally_reserved":0,"allocated":119922688,"data_stored":119637113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.41999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497216,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874624,"statfs":{"total":96636764160,"available":96127614976,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":508686028},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.435},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.47899999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313376,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119005184,"data_stored":119001705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:45.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:46.103 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:46.103 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:46.125 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-04-22T08:39:44.475266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490972,"kb_used_data":178748,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281624548,"statfs":{"total":289910292480,"available":288383537152,"internally_reserved":0,"allocated":183037952,"data_stored":182170294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.739066"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27630,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:34.855329+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:34.855329+0000","last_peered":"2024-04-22T08:39:34.855329+0000","last_clean":"2024-04-22T08:39:34.855329+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:34.855329+0000","last_undegraded":"2024-04-22T08:39:34.855329+0000","last_fullsized":"2024-04-22T08:39:34.855329+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":24884,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:42.767571+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:42.767571+0000","last_peered":"2024-04-22T08:39:42.767571+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:42.767571+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181276672,"data_stored":181248840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117148,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473600,"kb_used_data":117112,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93898240,"statfs":{"total":96636764160,"available":96151797760,"internally_reserved":0,"allocated":119922688,"data_stored":119637113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.41999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920910,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505408,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866432,"statfs":{"total":96636764160,"available":96119226368,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.308}]}]},{"osd":0,"up_from":21,"seq":90194313377,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.222},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119005184,"data_stored":119001705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:46.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:46.474 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:46.474 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:46.496 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-04-22T08:39:46.476141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490972,"kb_used_data":178748,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281624548,"statfs":{"total":289910292480,"available":288383537152,"internally_reserved":0,"allocated":183037952,"data_stored":182170294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002890"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27630,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:34.855329+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:34.855329+0000","last_peered":"2024-04-22T08:39:34.855329+0000","last_clean":"2024-04-22T08:39:34.855329+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:34.855329+0000","last_undegraded":"2024-04-22T08:39:34.855329+0000","last_fullsized":"2024-04-22T08:39:34.855329+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":24884,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:42.767571+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:42.767571+0000","last_peered":"2024-04-22T08:39:42.767571+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:42.767571+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181276672,"data_stored":181248840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117148,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473600,"kb_used_data":117112,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93898240,"statfs":{"total":96636764160,"available":96151797760,"internally_reserved":0,"allocated":119922688,"data_stored":119637113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.41999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920910,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505408,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866432,"statfs":{"total":96636764160,"available":96119226368,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.308}]}]},{"osd":0,"up_from":21,"seq":90194313377,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.222},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119005184,"data_stored":119001705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:46.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:46.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20901, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735551+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:31.735551+0000', 'last_peered': '2024-04-22T08:39:31.735551+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:31.735551+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:46.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:46.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21082, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.733929+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:31.733929+0000', 'last_peered': '2024-04-22T08:39:31.733929+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:31.733929+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:46.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:39:46.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27630, 'reported_epoch': 51, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:39:34.855329+0000', 'last_change': '2024-04-22T08:39:36.837655+0000', 'last_active': '2024-04-22T08:39:34.855329+0000', 'last_peered': '2024-04-22T08:39:34.855329+0000', 'last_clean': '2024-04-22T08:39:34.855329+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:34.855329+0000', 'last_undegraded': '2024-04-22T08:39:34.855329+0000', 'last_fullsized': '2024-04-22T08:39:34.855329+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:46.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:46.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 24884, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:42.767571+0000', 'last_change': '2024-04-22T08:39:31.310909+0000', 'last_active': '2024-04-22T08:39:42.767571+0000', 'last_peered': '2024-04-22T08:39:42.767571+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:39:42.767571+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2324, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1399}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:46.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:46.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20437, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735676+0000', 'last_change': '2024-04-22T08:27:56.757502+0000', 'last_active': '2024-04-22T08:39:31.735676+0000', 'last_peered': '2024-04-22T08:39:31.735676+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:31.735676+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 247, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:46.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:46.860 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:46.860 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11834/26 objects degraded (45515.385%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1526755328,"bytes_avail":288383537152,"bytes_total":289910292480,"degraded_objects":11834,"degraded_total":26,"degraded_ratio":455.15384615384608,"recovering_objects_per_sec":150,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:46.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:46.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:47.198 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:47.199 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:47.219 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-04-22T08:39:46.476141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490972,"kb_used_data":178748,"kb_used_omap":12,"kb_used_meta":1312179,"kb_avail":281624548,"statfs":{"total":289910292480,"available":288383537152,"internally_reserved":0,"allocated":183037952,"data_stored":182170294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1343671406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1807,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002890"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27630,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:34.855329+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:34.855329+0000","last_peered":"2024-04-22T08:39:34.855329+0000","last_clean":"2024-04-22T08:39:34.855329+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:34.855329+0000","last_undegraded":"2024-04-22T08:39:34.855329+0000","last_fullsized":"2024-04-22T08:39:34.855329+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":24884,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:42.767571+0000","last_change":"2024-04-22T08:39:31.310909+0000","last_active":"2024-04-22T08:39:42.767571+0000","last_peered":"2024-04-22T08:39:42.767571+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:42.767571+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20437,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735676+0000","last_change":"2024-04-22T08:27:56.757502+0000","last_active":"2024-04-22T08:39:31.735676+0000","last_peered":"2024-04-22T08:39:31.735676+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:31.735676+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33773,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181276672,"data_stored":181248840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117148,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473600,"kb_used_data":117112,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93898240,"statfs":{"total":96636764160,"available":96151797760,"internally_reserved":0,"allocated":119922688,"data_stored":119637113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.312}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.41999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920910,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505408,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866432,"statfs":{"total":96636764160,"available":96119226368,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":517074636},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.308}]}]},{"osd":0,"up_from":21,"seq":90194313377,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.222},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119005184,"data_stored":119001705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:50.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:50.572 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:50.573 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:50.593 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-22T08:39:50.477043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1484476,"kb_used_data":176348,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281631044,"statfs":{"total":289910292480,"available":288390189056,"internally_reserved":0,"allocated":180580352,"data_stored":179712694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002141"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27631,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:36.837655+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:36.837655+0000","last_peered":"2024-04-22T08:39:36.837655+0000","last_clean":"2024-04-22T08:39:36.837655+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:36.837655+0000","last_undegraded":"2024-04-22T08:39:36.837655+0000","last_fullsized":"2024-04-22T08:39:36.837655+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":27685,"reported_epoch":51,"state":"active+undersized+remapped","last_fresh":"2024-04-22T08:39:47.299524+0000","last_change":"2024-04-22T08:39:47.299711+0000","last_active":"2024-04-22T08:39:47.299524+0000","last_peered":"2024-04-22T08:39:47.299524+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:47.299524+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20492,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.765638+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:47.765638+0000","last_peered":"2024-04-22T08:39:47.765638+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:47.765638+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178806784,"data_stored":178789389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117149,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471200,"kb_used_data":114712,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93900640,"statfs":{"total":96636764160,"available":96154255360,"internally_reserved":0,"allocated":117465088,"data_stored":117179513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.318}]}]},{"osd":1,"up_from":33,"seq":141733920911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501312,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870528,"statfs":{"total":96636764160,"available":96123420672,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.23400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313378,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116547584,"data_stored":116544105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:50.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:50.933 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:50.933 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:50.953 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-22T08:39:50.477043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1484476,"kb_used_data":176348,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281631044,"statfs":{"total":289910292480,"available":288390189056,"internally_reserved":0,"allocated":180580352,"data_stored":179712694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002141"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27631,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:36.837655+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:36.837655+0000","last_peered":"2024-04-22T08:39:36.837655+0000","last_clean":"2024-04-22T08:39:36.837655+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:36.837655+0000","last_undegraded":"2024-04-22T08:39:36.837655+0000","last_fullsized":"2024-04-22T08:39:36.837655+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":27685,"reported_epoch":51,"state":"active+undersized+remapped","last_fresh":"2024-04-22T08:39:47.299524+0000","last_change":"2024-04-22T08:39:47.299711+0000","last_active":"2024-04-22T08:39:47.299524+0000","last_peered":"2024-04-22T08:39:47.299524+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:47.299524+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20492,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.765638+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:47.765638+0000","last_peered":"2024-04-22T08:39:47.765638+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:47.765638+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178806784,"data_stored":178789389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117149,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471200,"kb_used_data":114712,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93900640,"statfs":{"total":96636764160,"available":96154255360,"internally_reserved":0,"allocated":117465088,"data_stored":117179513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.318}]}]},{"osd":1,"up_from":33,"seq":141733920911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501312,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870528,"statfs":{"total":96636764160,"available":96123420672,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.23400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313378,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116547584,"data_stored":116544105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20901, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.735551+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:31.735551+0000', 'last_peered': '2024-04-22T08:39:31.735551+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:31.735551+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21082, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:31.733929+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:31.733929+0000', 'last_peered': '2024-04-22T08:39:31.733929+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:31.733929+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:39:50.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27631, 'reported_epoch': 51, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:39:36.837655+0000', 'last_change': '2024-04-22T08:39:36.837655+0000', 'last_active': '2024-04-22T08:39:36.837655+0000', 'last_peered': '2024-04-22T08:39:36.837655+0000', 'last_clean': '2024-04-22T08:39:36.837655+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:36.837655+0000', 'last_undegraded': '2024-04-22T08:39:36.837655+0000', 'last_fullsized': '2024-04-22T08:39:36.837655+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:50.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-04-22T08:39:50.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'20142", 'reported_seq': 27685, 'reported_epoch': 51, 'state': 'active+undersized+remapped', 'last_fresh': '2024-04-22T08:39:47.299524+0000', 'last_change': '2024-04-22T08:39:47.299711+0000', 'last_active': '2024-04-22T08:39:47.299524+0000', 'last_peered': '2024-04-22T08:39:47.299524+0000', 'last_clean': '2024-04-22T08:27:54.059442+0000', 'last_became_active': '2024-04-22T08:27:56.679255+0000', 'last_became_peered': '2024-04-22T08:27:56.679255+0000', 'last_unstale': '2024-04-22T08:39:47.299524+0000', 'last_undegraded': '2024-04-22T08:27:56.244421+0000', 'last_fullsized': '2024-04-22T08:27:56.095026+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20141, 'num_write_kb': 40281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3724, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '1', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:50.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:50.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 20492, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.765638+0000', 'last_change': '2024-04-22T08:39:47.300333+0000', 'last_active': '2024-04-22T08:39:47.765638+0000', 'last_peered': '2024-04-22T08:39:47.765638+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:47.765638+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3566, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 274, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:50.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10407/26 objects degraded (40026.923%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1520103424,"bytes_avail":288390189056,"bytes_total":289910292480,"degraded_objects":10407,"degraded_total":26,"degraded_ratio":400.26923076923077,"recovering_objects_per_sec":309,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2474,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:51.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:51.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:52.093 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:52.094 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:52.113 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-22T08:39:50.477043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1484476,"kb_used_data":176348,"kb_used_omap":12,"kb_used_meta":1308083,"kb_avail":281631044,"statfs":{"total":289910292480,"available":288390189056,"internally_reserved":0,"allocated":180580352,"data_stored":179712694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1339477102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002141"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20901,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.735551+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:31.735551+0000","last_peered":"2024-04-22T08:39:31.735551+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:31.735551+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20365,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739848+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:31.739848+0000","last_peered":"2024-04-22T08:39:31.739848+0000","last_clean":"2024-04-22T08:39:31.739848+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:31.739848+0000","last_undegraded":"2024-04-22T08:39:31.739848+0000","last_fullsized":"2024-04-22T08:39:31.739848+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20417,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733517+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:31.733517+0000","last_peered":"2024-04-22T08:39:31.733517+0000","last_clean":"2024-04-22T08:39:31.733517+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:31.733517+0000","last_undegraded":"2024-04-22T08:39:31.733517+0000","last_fullsized":"2024-04-22T08:39:31.733517+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.739855+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:31.739855+0000","last_peered":"2024-04-22T08:39:31.739855+0000","last_clean":"2024-04-22T08:39:31.739855+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:31.739855+0000","last_undegraded":"2024-04-22T08:39:31.739855+0000","last_fullsized":"2024-04-22T08:39:31.739855+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20075,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.734793+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:31.734793+0000","last_peered":"2024-04-22T08:39:31.734793+0000","last_clean":"2024-04-22T08:39:31.734793+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:31.734793+0000","last_undegraded":"2024-04-22T08:39:31.734793+0000","last_fullsized":"2024-04-22T08:39:31.734793+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28001,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733315+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:31.733315+0000","last_peered":"2024-04-22T08:39:31.733315+0000","last_clean":"2024-04-22T08:39:31.733315+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:31.733315+0000","last_undegraded":"2024-04-22T08:39:31.733315+0000","last_fullsized":"2024-04-22T08:39:31.733315+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27618,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733105+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:31.733105+0000","last_peered":"2024-04-22T08:39:31.733105+0000","last_clean":"2024-04-22T08:39:31.733105+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:31.733105+0000","last_undegraded":"2024-04-22T08:39:31.733105+0000","last_fullsized":"2024-04-22T08:39:31.733105+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19968,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733103+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:31.733103+0000","last_peered":"2024-04-22T08:39:31.733103+0000","last_clean":"2024-04-22T08:39:31.733103+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:31.733103+0000","last_undegraded":"2024-04-22T08:39:31.733103+0000","last_fullsized":"2024-04-22T08:39:31.733103+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.732809+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:31.732809+0000","last_peered":"2024-04-22T08:39:31.732809+0000","last_clean":"2024-04-22T08:39:31.732809+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:31.732809+0000","last_undegraded":"2024-04-22T08:39:31.732809+0000","last_fullsized":"2024-04-22T08:39:31.732809+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20693,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.839760+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:32.839760+0000","last_peered":"2024-04-22T08:39:32.839760+0000","last_clean":"2024-04-22T08:39:32.839760+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:32.839760+0000","last_undegraded":"2024-04-22T08:39:32.839760+0000","last_fullsized":"2024-04-22T08:39:32.839760+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739200+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:32.739200+0000","last_peered":"2024-04-22T08:39:32.739200+0000","last_clean":"2024-04-22T08:39:32.739200+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:32.739200+0000","last_undegraded":"2024-04-22T08:39:32.739200+0000","last_fullsized":"2024-04-22T08:39:32.739200+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21082,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:31.733929+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:31.733929+0000","last_peered":"2024-04-22T08:39:31.733929+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:31.733929+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733329+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:31.733329+0000","last_peered":"2024-04-22T08:39:31.733329+0000","last_clean":"2024-04-22T08:39:31.733329+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:31.733329+0000","last_undegraded":"2024-04-22T08:39:31.733329+0000","last_fullsized":"2024-04-22T08:39:31.733329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27608,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.739279+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:32.739279+0000","last_peered":"2024-04-22T08:39:32.739279+0000","last_clean":"2024-04-22T08:39:32.739279+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:32.739279+0000","last_undegraded":"2024-04-22T08:39:32.739279+0000","last_fullsized":"2024-04-22T08:39:32.739279+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733450+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:31.733450+0000","last_peered":"2024-04-22T08:39:31.733450+0000","last_clean":"2024-04-22T08:39:31.733450+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:31.733450+0000","last_undegraded":"2024-04-22T08:39:31.733450+0000","last_fullsized":"2024-04-22T08:39:31.733450+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27631,"reported_epoch":51,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:36.837655+0000","last_change":"2024-04-22T08:39:36.837655+0000","last_active":"2024-04-22T08:39:36.837655+0000","last_peered":"2024-04-22T08:39:36.837655+0000","last_clean":"2024-04-22T08:39:36.837655+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:36.837655+0000","last_undegraded":"2024-04-22T08:39:36.837655+0000","last_fullsized":"2024-04-22T08:39:36.837655+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":27685,"reported_epoch":51,"state":"active+undersized+remapped","last_fresh":"2024-04-22T08:39:47.299524+0000","last_change":"2024-04-22T08:39:47.299711+0000","last_active":"2024-04-22T08:39:47.299524+0000","last_peered":"2024-04-22T08:39:47.299524+0000","last_clean":"2024-04-22T08:27:54.059442+0000","last_became_active":"2024-04-22T08:27:56.679255+0000","last_became_peered":"2024-04-22T08:27:56.679255+0000","last_unstale":"2024-04-22T08:39:47.299524+0000","last_undegraded":"2024-04-22T08:27:56.244421+0000","last_fullsized":"2024-04-22T08:27:56.095026+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733445+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:31.733445+0000","last_peered":"2024-04-22T08:39:31.733445+0000","last_clean":"2024-04-22T08:39:31.733445+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:31.733445+0000","last_undegraded":"2024-04-22T08:39:31.733445+0000","last_fullsized":"2024-04-22T08:39:31.733445+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20492,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.765638+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:47.765638+0000","last_peered":"2024-04-22T08:39:47.765638+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:47.765638+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-22T08:39:32.740475+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:32.740475+0000","last_peered":"2024-04-22T08:39:32.740475+0000","last_clean":"2024-04-22T08:39:32.740475+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:32.740475+0000","last_undegraded":"2024-04-22T08:39:32.740475+0000","last_fullsized":"2024-04-22T08:39:32.740475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20659,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733572+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:31.733572+0000","last_peered":"2024-04-22T08:39:31.733572+0000","last_clean":"2024-04-22T08:39:31.733572+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:31.733572+0000","last_undegraded":"2024-04-22T08:39:31.733572+0000","last_fullsized":"2024-04-22T08:39:31.733572+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733657+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:31.733657+0000","last_peered":"2024-04-22T08:39:31.733657+0000","last_clean":"2024-04-22T08:39:31.733657+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:31.733657+0000","last_undegraded":"2024-04-22T08:39:31.733657+0000","last_fullsized":"2024-04-22T08:39:31.733657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733752+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:31.733752+0000","last_peered":"2024-04-22T08:39:31.733752+0000","last_clean":"2024-04-22T08:39:31.733752+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:31.733752+0000","last_undegraded":"2024-04-22T08:39:31.733752+0000","last_fullsized":"2024-04-22T08:39:31.733752+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20733,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-22T08:39:31.733470+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:31.733470+0000","last_peered":"2024-04-22T08:39:31.733470+0000","last_clean":"2024-04-22T08:39:31.733470+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:31.733470+0000","last_undegraded":"2024-04-22T08:39:31.733470+0000","last_fullsized":"2024-04-22T08:39:31.733470+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35200,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178806784,"data_stored":178789389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117149,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471200,"kb_used_data":114712,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93900640,"statfs":{"total":96636764160,"available":96154255360,"internally_reserved":0,"allocated":117465088,"data_stored":117179513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.318}]}]},{"osd":1,"up_from":33,"seq":141733920911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501312,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870528,"statfs":{"total":96636764160,"available":96123420672,"internally_reserved":0,"allocated":450560,"data_stored":147590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":512880332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.23400000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313378,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511964,"kb_used_data":61196,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859876,"statfs":{"total":96636764160,"available":96112513024,"internally_reserved":0,"allocated":62664704,"data_stored":62385591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116547584,"data_stored":116544105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:55.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:55.599 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:55.599 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:55.619 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-04-22T08:39:54.477771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1504028,"kb_used_data":173748,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281611492,"statfs":{"total":289910292480,"available":288370167808,"internally_reserved":0,"allocated":177917952,"data_stored":177028346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801461+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:47.801461+0000","last_peered":"2024-04-22T08:39:47.801461+0000","last_clean":"2024-04-22T08:39:47.801461+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:47.801461+0000","last_undegraded":"2024-04-22T08:39:47.801461+0000","last_fullsized":"2024-04-22T08:39:47.801461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":22030,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:52.774719+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:52.774719+0000","last_peered":"2024-04-22T08:39:52.774719+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:52.774719+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175984640,"data_stored":175967245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117150,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468588,"kb_used_data":112092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93903252,"statfs":{"total":96636764160,"available":96156930048,"internally_reserved":0,"allocated":114782208,"data_stored":114485838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.54300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523460,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848380,"statfs":{"total":96636764160,"available":96100741120,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313379,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113725440,"data_stored":113721961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:55.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:55.943 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:55.944 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:55.965 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-04-22T08:39:54.477771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1504028,"kb_used_data":173748,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281611492,"statfs":{"total":289910292480,"available":288370167808,"internally_reserved":0,"allocated":177917952,"data_stored":177028346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801461+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:47.801461+0000","last_peered":"2024-04-22T08:39:47.801461+0000","last_clean":"2024-04-22T08:39:47.801461+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:47.801461+0000","last_undegraded":"2024-04-22T08:39:47.801461+0000","last_fullsized":"2024-04-22T08:39:47.801461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":22030,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:52.774719+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:52.774719+0000","last_peered":"2024-04-22T08:39:52.774719+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:52.774719+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175984640,"data_stored":175967245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117150,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468588,"kb_used_data":112092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93903252,"statfs":{"total":96636764160,"available":96156930048,"internally_reserved":0,"allocated":114782208,"data_stored":114485838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.54300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523460,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848380,"statfs":{"total":96636764160,"available":96100741120,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313379,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113725440,"data_stored":113721961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20903, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.805269+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:47.805269+0000', 'last_peered': '2024-04-22T08:39:47.805269+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:47.805269+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21084, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.803924+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:47.803924+0000', 'last_peered': '2024-04-22T08:39:47.803924+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:47.803924+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27634, 'reported_epoch': 53, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-22T08:39:48.810204+0000', 'last_change': '2024-04-22T08:39:48.932096+0000', 'last_active': '2024-04-22T08:39:48.810204+0000', 'last_peered': '2024-04-22T08:39:48.810204+0000', 'last_clean': '2024-04-22T08:39:48.810204+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:48.810204+0000', 'last_undegraded': '2024-04-22T08:39:48.810204+0000', 'last_fullsized': '2024-04-22T08:39:48.810204+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:39:55.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 22030, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:52.774719+0000', 'last_change': '2024-04-22T08:39:47.300333+0000', 'last_active': '2024-04-22T08:39:52.774719+0000', 'last_peered': '2024-04-22T08:39:52.774719+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:52.774719+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2798, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1042, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:39:55.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:39:56.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:56.389 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9639/26 objects degraded (37073.077%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":831,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1540124672,"bytes_avail":288370167808,"bytes_total":289910292480,"degraded_objects":9639,"degraded_total":26,"degraded_ratio":370.73076923076923,"recovering_objects_per_sec":274,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:39:56.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:39:56.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:39:56.699 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:39:56.700 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:39:56.715 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-04-22T08:39:56.478514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1504028,"kb_used_data":173748,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281611492,"statfs":{"total":289910292480,"available":288370167808,"internally_reserved":0,"allocated":177917952,"data_stored":177028346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.678570"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801461+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:47.801461+0000","last_peered":"2024-04-22T08:39:47.801461+0000","last_clean":"2024-04-22T08:39:47.801461+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:47.801461+0000","last_undegraded":"2024-04-22T08:39:47.801461+0000","last_fullsized":"2024-04-22T08:39:47.801461+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":22030,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:52.774719+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:52.774719+0000","last_peered":"2024-04-22T08:39:52.774719+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:52.774719+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2798}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35968,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175984640,"data_stored":175967245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117150,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468588,"kb_used_data":112092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93903252,"statfs":{"total":96636764160,"available":96156930048,"internally_reserved":0,"allocated":114782208,"data_stored":114485838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.54300000000000004}]}]},{"osd":1,"up_from":33,"seq":141733920912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523460,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848380,"statfs":{"total":96636764160,"available":96100741120,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.317}]}]},{"osd":0,"up_from":21,"seq":90194313379,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.503}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113725440,"data_stored":113721961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:39:59.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:00.054 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:00.055 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:00.077 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-22T08:39:58.478808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1492828,"kb_used_data":170748,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281622692,"statfs":{"total":289910292480,"available":288381636608,"internally_reserved":0,"allocated":174845952,"data_stored":173956346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002340"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":23940,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:57.774440+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:57.774440+0000","last_peered":"2024-04-22T08:39:57.774440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:57.774440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172912640,"data_stored":172895245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117151,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465580,"kb_used_data":109092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93906260,"statfs":{"total":96636764160,"available":96160010240,"internally_reserved":0,"allocated":111710208,"data_stored":111413838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.308},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.40799999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856572,"statfs":{"total":96636764160,"available":96109129728,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313380,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.247},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110653440,"data_stored":110649961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:00.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:00.430 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:00.448 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-22T08:39:58.478808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1492828,"kb_used_data":170748,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281622692,"statfs":{"total":289910292480,"available":288381636608,"internally_reserved":0,"allocated":174845952,"data_stored":173956346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002340"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":23940,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:57.774440+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:57.774440+0000","last_peered":"2024-04-22T08:39:57.774440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:57.774440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172912640,"data_stored":172895245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117151,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465580,"kb_used_data":109092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93906260,"statfs":{"total":96636764160,"available":96160010240,"internally_reserved":0,"allocated":111710208,"data_stored":111413838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.308},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.40799999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856572,"statfs":{"total":96636764160,"available":96109129728,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313380,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.247},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110653440,"data_stored":110649961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:00.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:00.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20903, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.805269+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:47.805269+0000', 'last_peered': '2024-04-22T08:39:47.805269+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:47.805269+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:00.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:00.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21084, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.803924+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:47.803924+0000', 'last_peered': '2024-04-22T08:39:47.803924+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:47.803924+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:00.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:00.450 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27634, 'reported_epoch': 53, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-22T08:39:48.810204+0000', 'last_change': '2024-04-22T08:39:48.932096+0000', 'last_active': '2024-04-22T08:39:48.810204+0000', 'last_peered': '2024-04-22T08:39:48.810204+0000', 'last_clean': '2024-04-22T08:39:48.810204+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:48.810204+0000', 'last_undegraded': '2024-04-22T08:39:48.810204+0000', 'last_fullsized': '2024-04-22T08:39:48.810204+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:00.450 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:40:00.450 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 23940, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:57.774440+0000', 'last_change': '2024-04-22T08:39:47.300333+0000', 'last_active': '2024-04-22T08:39:57.774440+0000', 'last_peered': '2024-04-22T08:39:57.774440+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:39:57.774440+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1843, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1997, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:00.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:00.846 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:00.847 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8684/26 objects degraded (33400.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1528655872,"bytes_avail":288381636608,"bytes_total":289910292480,"degraded_objects":8684,"degraded_total":26,"degraded_ratio":334,"recovering_objects_per_sec":172,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:00.866 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:00.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:01.170 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:01.188 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-04-22T08:40:00.479208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1492828,"kb_used_data":170748,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281622692,"statfs":{"total":289910292480,"available":288381636608,"internally_reserved":0,"allocated":174845952,"data_stored":173956346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.671011"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27634,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-22T08:39:48.810204+0000","last_change":"2024-04-22T08:39:48.932096+0000","last_active":"2024-04-22T08:39:48.810204+0000","last_peered":"2024-04-22T08:39:48.810204+0000","last_clean":"2024-04-22T08:39:48.810204+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:48.810204+0000","last_undegraded":"2024-04-22T08:39:48.810204+0000","last_fullsized":"2024-04-22T08:39:48.810204+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":23940,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:57.774440+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:39:57.774440+0000","last_peered":"2024-04-22T08:39:57.774440+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:39:57.774440+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20661,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802674+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:47.802674+0000","last_peered":"2024-04-22T08:39:47.802674+0000","last_clean":"2024-04-22T08:39:47.802674+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:47.802674+0000","last_undegraded":"2024-04-22T08:39:47.802674+0000","last_fullsized":"2024-04-22T08:39:47.802674+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36923,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172912640,"data_stored":172895245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117151,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465580,"kb_used_data":109092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93906260,"statfs":{"total":96636764160,"available":96160010240,"internally_reserved":0,"allocated":111710208,"data_stored":111413838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.308},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.40799999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856572,"statfs":{"total":96636764160,"available":96109129728,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.28000000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313380,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.247},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110653440,"data_stored":110649961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:04.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:04.539 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:04.557 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-04-22T08:40:04.479803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1502116,"kb_used_data":167748,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281613404,"statfs":{"total":289910292480,"available":288372125696,"internally_reserved":0,"allocated":171773952,"data_stored":170884346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002450"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27636,"reported_epoch":53,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:58.023055+0000","last_change":"2024-04-22T08:39:58.023055+0000","last_active":"2024-04-22T08:39:58.023055+0000","last_peered":"2024-04-22T08:39:58.023055+0000","last_clean":"2024-04-22T08:39:58.023055+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:58.023055+0000","last_undegraded":"2024-04-22T08:39:58.023055+0000","last_fullsized":"2024-04-22T08:39:58.023055+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":26618,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:02.777524+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:40:02.777524+0000","last_peered":"2024-04-22T08:40:02.777524+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:40:02.777524+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20662,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.815343+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:48.815343+0000","last_peered":"2024-04-22T08:39:48.815343+0000","last_clean":"2024-04-22T08:39:48.815343+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:48.815343+0000","last_undegraded":"2024-04-22T08:39:48.815343+0000","last_fullsized":"2024-04-22T08:39:48.815343+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169840640,"data_stored":169823245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117152,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462580,"kb_used_data":106092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93909260,"statfs":{"total":96636764160,"available":96163082240,"internally_reserved":0,"allocated":108638208,"data_stored":108341838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.59999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527556,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844284,"statfs":{"total":96636764160,"available":96096546816,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.307}]}]},{"osd":0,"up_from":21,"seq":90194313381,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.28599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107581440,"data_stored":107577961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:04.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:04.865 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:04.884 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-04-22T08:40:04.479803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1502116,"kb_used_data":167748,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281613404,"statfs":{"total":289910292480,"available":288372125696,"internally_reserved":0,"allocated":171773952,"data_stored":170884346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002450"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27636,"reported_epoch":53,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:58.023055+0000","last_change":"2024-04-22T08:39:58.023055+0000","last_active":"2024-04-22T08:39:58.023055+0000","last_peered":"2024-04-22T08:39:58.023055+0000","last_clean":"2024-04-22T08:39:58.023055+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:58.023055+0000","last_undegraded":"2024-04-22T08:39:58.023055+0000","last_fullsized":"2024-04-22T08:39:58.023055+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":26618,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:02.777524+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:40:02.777524+0000","last_peered":"2024-04-22T08:40:02.777524+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:40:02.777524+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20662,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.815343+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:48.815343+0000","last_peered":"2024-04-22T08:39:48.815343+0000","last_clean":"2024-04-22T08:39:48.815343+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:48.815343+0000","last_undegraded":"2024-04-22T08:39:48.815343+0000","last_fullsized":"2024-04-22T08:39:48.815343+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169840640,"data_stored":169823245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117152,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462580,"kb_used_data":106092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93909260,"statfs":{"total":96636764160,"available":96163082240,"internally_reserved":0,"allocated":108638208,"data_stored":108341838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.59999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527556,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844284,"statfs":{"total":96636764160,"available":96096546816,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.307}]}]},{"osd":0,"up_from":21,"seq":90194313381,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.28599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107581440,"data_stored":107577961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20903, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.805269+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:39:47.805269+0000', 'last_peered': '2024-04-22T08:39:47.805269+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:39:47.805269+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 21084, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:39:47.803924+0000', 'last_change': '2024-04-22T08:27:56.759222+0000', 'last_active': '2024-04-22T08:39:47.803924+0000', 'last_peered': '2024-04-22T08:39:47.803924+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:39:47.803924+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3369}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27636, 'reported_epoch': 53, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:39:58.023055+0000', 'last_change': '2024-04-22T08:39:58.023055+0000', 'last_active': '2024-04-22T08:39:58.023055+0000', 'last_peered': '2024-04-22T08:39:58.023055+0000', 'last_clean': '2024-04-22T08:39:58.023055+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:39:58.023055+0000', 'last_undegraded': '2024-04-22T08:39:58.023055+0000', 'last_fullsized': '2024-04-22T08:39:58.023055+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-04-22T08:40:04.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "41'19750", 'reported_seq': 26618, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:02.777524+0000', 'last_change': '2024-04-22T08:39:47.300333+0000', 'last_active': '2024-04-22T08:40:02.777524+0000', 'last_peered': '2024-04-22T08:40:02.777524+0000', 'last_clean': '2024-04-22T08:27:54.089760+0000', 'last_became_active': '2024-04-22T08:27:56.747228+0000', 'last_became_peered': '2024-04-22T08:27:56.747228+0000', 'last_unstale': '2024-04-22T08:40:02.777524+0000', 'last_undegraded': '2024-04-22T08:27:56.230200+0000', 'last_fullsized': '2024-04-22T08:27:56.094671+0000', 'mapping_epoch': 34, 'log_start': "37'9700", 'ondisk_log_start': "37'9700", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "32'3834", 'last_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:27:53.920523+0000', 'objects_scrubbed': 3761, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000', 'scrub_duration': 2.291677937, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 504, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15981, 'num_write_kb': 24440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3336, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:04.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:05.261 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:05.261 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7345/26 objects degraded (28250.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":840,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":54,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1538166784,"bytes_avail":288372125696,"bytes_total":289910292480,"degraded_objects":7345,"degraded_total":26,"degraded_ratio":282.5,"recovering_objects_per_sec":255,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3062,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:05.282 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:05.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:05.588 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:05.610 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-04-22T08:40:05.211124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1502116,"kb_used_data":167748,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281613404,"statfs":{"total":289910292480,"available":288372125696,"internally_reserved":0,"allocated":171773952,"data_stored":170884346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2294,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.733352"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20903,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.805269+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:39:47.805269+0000","last_peered":"2024-04-22T08:39:47.805269+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:39:47.805269+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814165+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:47.814165+0000","last_peered":"2024-04-22T08:39:47.814165+0000","last_clean":"2024-04-22T08:39:47.814165+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:47.814165+0000","last_undegraded":"2024-04-22T08:39:47.814165+0000","last_fullsized":"2024-04-22T08:39:47.814165+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20419,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802449+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:39:47.802449+0000","last_peered":"2024-04-22T08:39:47.802449+0000","last_clean":"2024-04-22T08:39:47.802449+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:39:47.802449+0000","last_undegraded":"2024-04-22T08:39:47.802449+0000","last_fullsized":"2024-04-22T08:39:47.802449+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814237+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:47.814237+0000","last_peered":"2024-04-22T08:39:47.814237+0000","last_clean":"2024-04-22T08:39:47.814237+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:47.814237+0000","last_undegraded":"2024-04-22T08:39:47.814237+0000","last_fullsized":"2024-04-22T08:39:47.814237+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803864+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:39:47.803864+0000","last_peered":"2024-04-22T08:39:47.803864+0000","last_clean":"2024-04-22T08:39:47.803864+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:39:47.803864+0000","last_undegraded":"2024-04-22T08:39:47.803864+0000","last_fullsized":"2024-04-22T08:39:47.803864+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28003,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802055+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:39:47.802055+0000","last_peered":"2024-04-22T08:39:47.802055+0000","last_clean":"2024-04-22T08:39:47.802055+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:39:47.802055+0000","last_undegraded":"2024-04-22T08:39:47.802055+0000","last_fullsized":"2024-04-22T08:39:47.802055+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27620,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802048+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:39:47.802048+0000","last_peered":"2024-04-22T08:39:47.802048+0000","last_clean":"2024-04-22T08:39:47.802048+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:39:47.802048+0000","last_undegraded":"2024-04-22T08:39:47.802048+0000","last_fullsized":"2024-04-22T08:39:47.802048+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19971,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810615+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:39:48.810615+0000","last_peered":"2024-04-22T08:39:48.810615+0000","last_clean":"2024-04-22T08:39:48.810615+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:39:48.810615+0000","last_undegraded":"2024-04-22T08:39:48.810615+0000","last_fullsized":"2024-04-22T08:39:48.810615+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.814545+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:39:48.814545+0000","last_peered":"2024-04-22T08:39:48.814545+0000","last_clean":"2024-04-22T08:39:48.814545+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:39:48.814545+0000","last_undegraded":"2024-04-22T08:39:48.814545+0000","last_fullsized":"2024-04-22T08:39:48.814545+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802305+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:39:47.802305+0000","last_peered":"2024-04-22T08:39:47.802305+0000","last_clean":"2024-04-22T08:39:47.802305+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:39:47.802305+0000","last_undegraded":"2024-04-22T08:39:47.802305+0000","last_fullsized":"2024-04-22T08:39:47.802305+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814806+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:47.814806+0000","last_peered":"2024-04-22T08:39:47.814806+0000","last_clean":"2024-04-22T08:39:47.814806+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:47.814806+0000","last_undegraded":"2024-04-22T08:39:47.814806+0000","last_fullsized":"2024-04-22T08:39:47.814806+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":21084,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:39:47.803924+0000","last_change":"2024-04-22T08:27:56.759222+0000","last_active":"2024-04-22T08:39:47.803924+0000","last_peered":"2024-04-22T08:39:47.803924+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:39:47.803924+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.803407+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:39:47.803407+0000","last_peered":"2024-04-22T08:39:47.803407+0000","last_clean":"2024-04-22T08:39:47.803407+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:39:47.803407+0000","last_undegraded":"2024-04-22T08:39:47.803407+0000","last_fullsized":"2024-04-22T08:39:47.803407+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814048+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:47.814048+0000","last_peered":"2024-04-22T08:39:47.814048+0000","last_clean":"2024-04-22T08:39:47.814048+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:47.814048+0000","last_undegraded":"2024-04-22T08:39:47.814048+0000","last_fullsized":"2024-04-22T08:39:47.814048+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.801996+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:39:47.801996+0000","last_peered":"2024-04-22T08:39:47.801996+0000","last_clean":"2024-04-22T08:39:47.801996+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:39:47.801996+0000","last_undegraded":"2024-04-22T08:39:47.801996+0000","last_fullsized":"2024-04-22T08:39:47.801996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27636,"reported_epoch":53,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:39:58.023055+0000","last_change":"2024-04-22T08:39:58.023055+0000","last_active":"2024-04-22T08:39:58.023055+0000","last_peered":"2024-04-22T08:39:58.023055+0000","last_clean":"2024-04-22T08:39:58.023055+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:39:58.023055+0000","last_undegraded":"2024-04-22T08:39:58.023055+0000","last_fullsized":"2024-04-22T08:39:58.023055+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20908,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.974226+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:39:48.974226+0000","last_peered":"2024-04-22T08:39:48.974226+0000","last_clean":"2024-04-22T08:39:48.974226+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:39:48.974226+0000","last_undegraded":"2024-04-22T08:39:48.974226+0000","last_fullsized":"2024-04-22T08:39:48.974226+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802382+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:39:47.802382+0000","last_peered":"2024-04-22T08:39:47.802382+0000","last_clean":"2024-04-22T08:39:47.802382+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:39:47.802382+0000","last_undegraded":"2024-04-22T08:39:47.802382+0000","last_fullsized":"2024-04-22T08:39:47.802382+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":26618,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:02.777524+0000","last_change":"2024-04-22T08:39:47.300333+0000","last_active":"2024-04-22T08:40:02.777524+0000","last_peered":"2024-04-22T08:40:02.777524+0000","last_clean":"2024-04-22T08:27:54.089760+0000","last_became_active":"2024-04-22T08:27:56.747228+0000","last_became_peered":"2024-04-22T08:27:56.747228+0000","last_unstale":"2024-04-22T08:40:02.777524+0000","last_undegraded":"2024-04-22T08:27:56.230200+0000","last_fullsized":"2024-04-22T08:27:56.094671+0000","mapping_epoch":34,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:50.412157+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.814468+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:47.814468+0000","last_peered":"2024-04-22T08:39:47.814468+0000","last_clean":"2024-04-22T08:39:47.814468+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:47.814468+0000","last_undegraded":"2024-04-22T08:39:47.814468+0000","last_fullsized":"2024-04-22T08:39:47.814468+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20662,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.815343+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:39:48.815343+0000","last_peered":"2024-04-22T08:39:48.815343+0000","last_clean":"2024-04-22T08:39:48.815343+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:39:48.815343+0000","last_undegraded":"2024-04-22T08:39:48.815343+0000","last_fullsized":"2024-04-22T08:39:48.815343+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802702+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:39:47.802702+0000","last_peered":"2024-04-22T08:39:47.802702+0000","last_clean":"2024-04-22T08:39:47.802702+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:39:47.802702+0000","last_undegraded":"2024-04-22T08:39:47.802702+0000","last_fullsized":"2024-04-22T08:39:47.802702+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802852+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:39:47.802852+0000","last_peered":"2024-04-22T08:39:47.802852+0000","last_clean":"2024-04-22T08:39:47.802852+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:39:47.802852+0000","last_undegraded":"2024-04-22T08:39:47.802852+0000","last_fullsized":"2024-04-22T08:39:47.802852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20735,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-22T08:39:47.802333+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:39:47.802333+0000","last_peered":"2024-04-22T08:39:47.802333+0000","last_clean":"2024-04-22T08:39:47.802333+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:39:47.802333+0000","last_undegraded":"2024-04-22T08:39:47.802333+0000","last_fullsized":"2024-04-22T08:39:47.802333+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38262,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169840640,"data_stored":169823245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117152,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462580,"kb_used_data":106092,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93909260,"statfs":{"total":96636764160,"available":96163082240,"internally_reserved":0,"allocated":108638208,"data_stored":108341838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.59999999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527556,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844284,"statfs":{"total":96636764160,"available":96096546816,"internally_reserved":0,"allocated":454656,"data_stored":151328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.307}]}]},{"osd":0,"up_from":21,"seq":90194313381,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511980,"kb_used_data":61212,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859860,"statfs":{"total":96636764160,"available":96112496640,"internally_reserved":0,"allocated":62681088,"data_stored":62391180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.28599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107581440,"data_stored":107577961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:08.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:08.979 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:08.980 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:09.001 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-22T08:40:08.480994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490968,"kb_used_data":164780,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281624552,"statfs":{"total":289910292480,"available":288383541248,"internally_reserved":0,"allocated":168734720,"data_stored":167824970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001785"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811897+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:48.811897+0000","last_peered":"2024-04-22T08:39:48.811897+0000","last_clean":"2024-04-22T08:39:48.811897+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:48.811897+0000","last_undegraded":"2024-04-22T08:39:48.811897+0000","last_fullsized":"2024-04-22T08:39:48.811897+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810886+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:48.810886+0000","last_peered":"2024-04-22T08:39:48.810886+0000","last_clean":"2024-04-22T08:39:48.810886+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:48.810886+0000","last_undegraded":"2024-04-22T08:39:48.810886+0000","last_fullsized":"2024-04-22T08:39:48.810886+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20079,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216532+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:05.216532+0000","last_peered":"2024-04-22T08:40:05.216532+0000","last_clean":"2024-04-22T08:40:05.216532+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:05.216532+0000","last_undegraded":"2024-04-22T08:40:05.216532+0000","last_fullsized":"2024-04-22T08:40:05.216532+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810504+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:48.810504+0000","last_peered":"2024-04-22T08:39:48.810504+0000","last_clean":"2024-04-22T08:39:48.810504+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:48.810504+0000","last_undegraded":"2024-04-22T08:39:48.810504+0000","last_fullsized":"2024-04-22T08:39:48.810504+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":22019,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:07.786584+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:07.786584+0000","last_peered":"2024-04-22T08:40:07.786584+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:07.786584+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2905}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27611,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810567+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:48.810567+0000","last_peered":"2024-04-22T08:39:48.810567+0000","last_clean":"2024-04-22T08:39:48.810567+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:48.810567+0000","last_undegraded":"2024-04-22T08:39:48.810567+0000","last_fullsized":"2024-04-22T08:39:48.810567+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27641,"reported_epoch":54,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:05.215164+0000","last_change":"2024-04-22T08:40:05.973003+0000","last_active":"2024-04-22T08:40:05.215164+0000","last_peered":"2024-04-22T08:40:05.215164+0000","last_clean":"2024-04-22T08:40:05.215164+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:05.215164+0000","last_undegraded":"2024-04-22T08:40:05.215164+0000","last_fullsized":"2024-04-22T08:40:05.215164+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811121+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:48.811121+0000","last_peered":"2024-04-22T08:39:48.811121+0000","last_clean":"2024-04-22T08:39:48.811121+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:48.811121+0000","last_undegraded":"2024-04-22T08:39:48.811121+0000","last_fullsized":"2024-04-22T08:39:48.811121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166760448,"data_stored":166750011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117153,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459592,"kb_used_data":103100,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93912248,"statfs":{"total":96636764160,"available":96166141952,"internally_reserved":0,"allocated":105574400,"data_stored":105272618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.29399999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519380,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852460,"statfs":{"total":96636764160,"available":96104919040,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.437}]}]},{"osd":0,"up_from":21,"seq":90194313382,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104509440,"data_stored":104505961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:09.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:09.366 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:09.387 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-22T08:40:08.480994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490968,"kb_used_data":164780,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281624552,"statfs":{"total":289910292480,"available":288383541248,"internally_reserved":0,"allocated":168734720,"data_stored":167824970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001785"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811897+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:48.811897+0000","last_peered":"2024-04-22T08:39:48.811897+0000","last_clean":"2024-04-22T08:39:48.811897+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:48.811897+0000","last_undegraded":"2024-04-22T08:39:48.811897+0000","last_fullsized":"2024-04-22T08:39:48.811897+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810886+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:48.810886+0000","last_peered":"2024-04-22T08:39:48.810886+0000","last_clean":"2024-04-22T08:39:48.810886+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:48.810886+0000","last_undegraded":"2024-04-22T08:39:48.810886+0000","last_fullsized":"2024-04-22T08:39:48.810886+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20079,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216532+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:05.216532+0000","last_peered":"2024-04-22T08:40:05.216532+0000","last_clean":"2024-04-22T08:40:05.216532+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:05.216532+0000","last_undegraded":"2024-04-22T08:40:05.216532+0000","last_fullsized":"2024-04-22T08:40:05.216532+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810504+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:48.810504+0000","last_peered":"2024-04-22T08:39:48.810504+0000","last_clean":"2024-04-22T08:39:48.810504+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:48.810504+0000","last_undegraded":"2024-04-22T08:39:48.810504+0000","last_fullsized":"2024-04-22T08:39:48.810504+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":22019,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:07.786584+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:07.786584+0000","last_peered":"2024-04-22T08:40:07.786584+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:07.786584+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2905}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27611,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810567+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:48.810567+0000","last_peered":"2024-04-22T08:39:48.810567+0000","last_clean":"2024-04-22T08:39:48.810567+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:48.810567+0000","last_undegraded":"2024-04-22T08:39:48.810567+0000","last_fullsized":"2024-04-22T08:39:48.810567+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27641,"reported_epoch":54,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:05.215164+0000","last_change":"2024-04-22T08:40:05.973003+0000","last_active":"2024-04-22T08:40:05.215164+0000","last_peered":"2024-04-22T08:40:05.215164+0000","last_clean":"2024-04-22T08:40:05.215164+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:05.215164+0000","last_undegraded":"2024-04-22T08:40:05.215164+0000","last_fullsized":"2024-04-22T08:40:05.215164+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811121+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:48.811121+0000","last_peered":"2024-04-22T08:39:48.811121+0000","last_clean":"2024-04-22T08:39:48.811121+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:48.811121+0000","last_undegraded":"2024-04-22T08:39:48.811121+0000","last_fullsized":"2024-04-22T08:39:48.811121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166760448,"data_stored":166750011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117153,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459592,"kb_used_data":103100,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93912248,"statfs":{"total":96636764160,"available":96166141952,"internally_reserved":0,"allocated":105574400,"data_stored":105272618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.29399999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519380,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852460,"statfs":{"total":96636764160,"available":96104919040,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.437}]}]},{"osd":0,"up_from":21,"seq":90194313382,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104509440,"data_stored":104505961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:09.388 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:09.388 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20905, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:05.217436+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:40:05.217436+0000', 'last_peered': '2024-04-22T08:40:05.217436+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:05.217436+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:09.389 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:09.389 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 22019, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:07.786584+0000', 'last_change': '2024-04-22T08:40:04.724914+0000', 'last_active': '2024-04-22T08:40:07.786584+0000', 'last_peered': '2024-04-22T08:40:07.786584+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:40:07.786584+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2907, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 869, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2905}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:09.389 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:09.389 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27641, 'reported_epoch': 54, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:05.215164+0000', 'last_change': '2024-04-22T08:40:05.973003+0000', 'last_active': '2024-04-22T08:40:05.215164+0000', 'last_peered': '2024-04-22T08:40:05.215164+0000', 'last_clean': '2024-04-22T08:40:05.215164+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:05.215164+0000', 'last_undegraded': '2024-04-22T08:40:05.215164+0000', 'last_fullsized': '2024-04-22T08:40:05.215164+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:09.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:09.803 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6375/26 objects degraded (24519.231%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":844,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1526751232,"bytes_avail":288383541248,"bytes_total":289910292480,"degraded_objects":6375,"degraded_total":26,"degraded_ratio":245.19230769230771,"recovering_objects_per_sec":288,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:09.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:09.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:10.138 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:10.158 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-22T08:40:08.480994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1490968,"kb_used_data":164780,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281624552,"statfs":{"total":289910292480,"available":288383541248,"internally_reserved":0,"allocated":168734720,"data_stored":167824970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001785"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20368,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811897+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:39:48.811897+0000","last_peered":"2024-04-22T08:39:48.811897+0000","last_clean":"2024-04-22T08:39:48.811897+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:39:48.811897+0000","last_undegraded":"2024-04-22T08:39:48.811897+0000","last_fullsized":"2024-04-22T08:39:48.811897+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810886+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:39:48.810886+0000","last_peered":"2024-04-22T08:39:48.810886+0000","last_clean":"2024-04-22T08:39:48.810886+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:39:48.810886+0000","last_undegraded":"2024-04-22T08:39:48.810886+0000","last_fullsized":"2024-04-22T08:39:48.810886+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20079,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216532+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:05.216532+0000","last_peered":"2024-04-22T08:40:05.216532+0000","last_clean":"2024-04-22T08:40:05.216532+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:05.216532+0000","last_undegraded":"2024-04-22T08:40:05.216532+0000","last_fullsized":"2024-04-22T08:40:05.216532+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810504+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:39:48.810504+0000","last_peered":"2024-04-22T08:39:48.810504+0000","last_clean":"2024-04-22T08:39:48.810504+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:39:48.810504+0000","last_undegraded":"2024-04-22T08:39:48.810504+0000","last_fullsized":"2024-04-22T08:39:48.810504+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":22019,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:07.786584+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:07.786584+0000","last_peered":"2024-04-22T08:40:07.786584+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:07.786584+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2905}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27611,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.810567+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:39:48.810567+0000","last_peered":"2024-04-22T08:39:48.810567+0000","last_clean":"2024-04-22T08:39:48.810567+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:39:48.810567+0000","last_undegraded":"2024-04-22T08:39:48.810567+0000","last_fullsized":"2024-04-22T08:39:48.810567+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27641,"reported_epoch":54,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:05.215164+0000","last_change":"2024-04-22T08:40:05.973003+0000","last_active":"2024-04-22T08:40:05.215164+0000","last_peered":"2024-04-22T08:40:05.215164+0000","last_clean":"2024-04-22T08:40:05.215164+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:05.215164+0000","last_undegraded":"2024-04-22T08:40:05.215164+0000","last_fullsized":"2024-04-22T08:40:05.215164+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T08:39:48.811121+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:39:48.811121+0000","last_peered":"2024-04-22T08:39:48.811121+0000","last_clean":"2024-04-22T08:39:48.811121+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:39:48.811121+0000","last_undegraded":"2024-04-22T08:39:48.811121+0000","last_fullsized":"2024-04-22T08:39:48.811121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166760448,"data_stored":166750011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117153,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459592,"kb_used_data":103100,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93912248,"statfs":{"total":96636764160,"available":96166141952,"internally_reserved":0,"allocated":105574400,"data_stored":105272618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.245},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.29399999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519380,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852460,"statfs":{"total":96636764160,"available":96104919040,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.437}]}]},{"osd":0,"up_from":21,"seq":90194313382,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104509440,"data_stored":104505961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:13.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:13.506 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:13.507 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:13.531 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-22T08:40:12.481762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487496,"kb_used_data":161312,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281628024,"statfs":{"total":289910292480,"available":288387096576,"internally_reserved":0,"allocated":165183488,"data_stored":164268305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":970,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001959"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20079,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216532+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:05.216532+0000","last_peered":"2024-04-22T08:40:05.216532+0000","last_clean":"2024-04-22T08:40:05.216532+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:05.216532+0000","last_undegraded":"2024-04-22T08:40:05.216532+0000","last_fullsized":"2024-04-22T08:40:05.216532+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":22019,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:07.786584+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:07.786584+0000","last_peered":"2024-04-22T08:40:07.786584+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:07.786584+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2905}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27646,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:08.904347+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:08.904347+0000","last_peered":"2024-04-22T08:40:08.904347+0000","last_clean":"2024-04-22T08:40:08.904347+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:08.904347+0000","last_undegraded":"2024-04-22T08:40:08.904347+0000","last_fullsized":"2024-04-22T08:40:08.904347+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163078144,"data_stored":163067707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117154,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456120,"kb_used_data":99632,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93915720,"statfs":{"total":96636764160,"available":96169697280,"internally_reserved":0,"allocated":102023168,"data_stored":101715953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.247},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.38}]}]},{"osd":1,"up_from":33,"seq":141733920915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519380,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852460,"statfs":{"total":96636764160,"available":96104919040,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.437}]}]},{"osd":0,"up_from":21,"seq":90194313382,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100827136,"data_stored":100823657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:13.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:13.848 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:13.849 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:13.866 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-22T08:40:12.481762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1487496,"kb_used_data":161312,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281628024,"statfs":{"total":289910292480,"available":288387096576,"internally_reserved":0,"allocated":165183488,"data_stored":164268305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":970,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001959"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20079,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216532+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:05.216532+0000","last_peered":"2024-04-22T08:40:05.216532+0000","last_clean":"2024-04-22T08:40:05.216532+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:05.216532+0000","last_undegraded":"2024-04-22T08:40:05.216532+0000","last_fullsized":"2024-04-22T08:40:05.216532+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":22019,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:07.786584+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:07.786584+0000","last_peered":"2024-04-22T08:40:07.786584+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:07.786584+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2905}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27646,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:08.904347+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:08.904347+0000","last_peered":"2024-04-22T08:40:08.904347+0000","last_clean":"2024-04-22T08:40:08.904347+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:08.904347+0000","last_undegraded":"2024-04-22T08:40:08.904347+0000","last_fullsized":"2024-04-22T08:40:08.904347+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39232,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163078144,"data_stored":163067707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117154,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456120,"kb_used_data":99632,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93915720,"statfs":{"total":96636764160,"available":96169697280,"internally_reserved":0,"allocated":102023168,"data_stored":101715953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.247},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.38}]}]},{"osd":1,"up_from":33,"seq":141733920915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519380,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852460,"statfs":{"total":96636764160,"available":96104919040,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.437}]}]},{"osd":0,"up_from":21,"seq":90194313382,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100827136,"data_stored":100823657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:13.867 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:13.867 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20905, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:05.217436+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:40:05.217436+0000', 'last_peered': '2024-04-22T08:40:05.217436+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:05.217436+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:13.867 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:13.867 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 22019, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:07.786584+0000', 'last_change': '2024-04-22T08:40:04.724914+0000', 'last_active': '2024-04-22T08:40:07.786584+0000', 'last_peered': '2024-04-22T08:40:07.786584+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:40:07.786584+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2907, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 869, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2905}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:13.868 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:13.868 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27646, 'reported_epoch': 55, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:08.904347+0000', 'last_change': '2024-04-22T08:40:07.906799+0000', 'last_active': '2024-04-22T08:40:08.904347+0000', 'last_peered': '2024-04-22T08:40:08.904347+0000', 'last_clean': '2024-04-22T08:40:08.904347+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:08.904347+0000', 'last_undegraded': '2024-04-22T08:40:08.904347+0000', 'last_fullsized': '2024-04-22T08:40:08.904347+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:13.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:14.288 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6375/26 objects degraded (24519.231%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":849,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1523195904,"bytes_avail":288387096576,"bytes_total":289910292480,"degraded_objects":6375,"degraded_total":26,"degraded_ratio":245.19230769230771,"recovering_objects_per_sec":121,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":970,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:14.306 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:14.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:14.626 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:14.627 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:14.645 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-04-22T08:40:14.482428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":161312,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":165183488,"data_stored":164268305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.271304"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":23853,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:12.791122+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:12.791122+0000","last_peered":"2024-04-22T08:40:12.791122+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:12.791122+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27646,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:08.904347+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:08.904347+0000","last_peered":"2024-04-22T08:40:08.904347+0000","last_clean":"2024-04-22T08:40:08.904347+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:08.904347+0000","last_undegraded":"2024-04-22T08:40:08.904347+0000","last_fullsized":"2024-04-22T08:40:08.904347+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163078144,"data_stored":163067707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117154,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456120,"kb_used_data":99632,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93915720,"statfs":{"total":96636764160,"available":96169697280,"internally_reserved":0,"allocated":102023168,"data_stored":101715953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.247},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.38}]}]},{"osd":1,"up_from":33,"seq":141733920916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527564,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844276,"statfs":{"total":96636764160,"available":96096538624,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313383,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100827136,"data_stored":100823657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:17.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:17.971 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:17.972 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:17.991 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-04-22T08:40:16.482826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":161312,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":165183488,"data_stored":164268305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.270381"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":23853,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:12.791122+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:12.791122+0000","last_peered":"2024-04-22T08:40:12.791122+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:12.791122+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27646,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:08.904347+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:08.904347+0000","last_peered":"2024-04-22T08:40:08.904347+0000","last_clean":"2024-04-22T08:40:08.904347+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:08.904347+0000","last_undegraded":"2024-04-22T08:40:08.904347+0000","last_fullsized":"2024-04-22T08:40:08.904347+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163078144,"data_stored":163067707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117154,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456120,"kb_used_data":99632,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93915720,"statfs":{"total":96636764160,"available":96169697280,"internally_reserved":0,"allocated":102023168,"data_stored":101715953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.247},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.38}]}]},{"osd":1,"up_from":33,"seq":141733920916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527564,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844276,"statfs":{"total":96636764160,"available":96096538624,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313383,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100827136,"data_stored":100823657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:17.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:18.316 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:18.316 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:18.335 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-04-22T08:40:16.482826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1495680,"kb_used_data":161312,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281619840,"statfs":{"total":289910292480,"available":288378716160,"internally_reserved":0,"allocated":165183488,"data_stored":164268305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.270381"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":23853,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:12.791122+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:12.791122+0000","last_peered":"2024-04-22T08:40:12.791122+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:12.791122+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27646,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:08.904347+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:08.904347+0000","last_peered":"2024-04-22T08:40:08.904347+0000","last_clean":"2024-04-22T08:40:08.904347+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:08.904347+0000","last_undegraded":"2024-04-22T08:40:08.904347+0000","last_fullsized":"2024-04-22T08:40:08.904347+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40149,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163078144,"data_stored":163067707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117154,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456120,"kb_used_data":99632,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93915720,"statfs":{"total":96636764160,"available":96169697280,"internally_reserved":0,"allocated":102023168,"data_stored":101715953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.247},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.28100000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.85199999999999998,"15min":1.073},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.72799999999999998,"5min":15.6,"15min":15.6},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.0469999999999999,"15min":1.0329999999999999},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.75900000000000001,"5min":6.6239999999999997,"15min":18.32},"last":0.38}]}]},{"osd":1,"up_from":33,"seq":141733920916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527564,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844276,"statfs":{"total":96636764160,"available":96096538624,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45000000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313383,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.1910000000000001,"15min":0.98199999999999998},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16},"max":{"1min":0.59599999999999997,"5min":11.368,"15min":11.368},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.95999999999999996,"15min":0.97499999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.77000000000000002,"5min":7.4400000000000004,"15min":19.658000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100827136,"data_stored":100823657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:18.336 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:18.336 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20905, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:05.217436+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:40:05.217436+0000', 'last_peered': '2024-04-22T08:40:05.217436+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:05.217436+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:18.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:18.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 23853, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:12.791122+0000', 'last_change': '2024-04-22T08:40:04.724914+0000', 'last_active': '2024-04-22T08:40:12.791122+0000', 'last_peered': '2024-04-22T08:40:12.791122+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:40:12.791122+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1786, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1988}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:18.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:18.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27646, 'reported_epoch': 55, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:08.904347+0000', 'last_change': '2024-04-22T08:40:07.906799+0000', 'last_active': '2024-04-22T08:40:08.904347+0000', 'last_peered': '2024-04-22T08:40:08.904347+0000', 'last_clean': '2024-04-22T08:40:08.904347+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:08.904347+0000', 'last_undegraded': '2024-04-22T08:40:08.904347+0000', 'last_fullsized': '2024-04-22T08:40:08.904347+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:18.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:18.726 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:18.726 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4192/26 objects degraded (16123.077%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":853,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1519501312,"bytes_avail":288390791168,"bytes_total":289910292480,"degraded_objects":4192,"degraded_total":26,"degraded_ratio":161.23076923076923,"recovering_objects_per_sec":262,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:18.744 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:18.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:19.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:19.051 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:19.067 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-04-22T08:40:18.483133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1483888,"kb_used_data":157712,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281631632,"statfs":{"total":289910292480,"available":288390791168,"internally_reserved":0,"allocated":161497088,"data_stored":160581905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002639"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":26385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:17.789841+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:17.789841+0000","last_peered":"2024-04-22T08:40:17.789841+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:17.789841+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3052,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-722}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27648,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:12.890546+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:12.890546+0000","last_peered":"2024-04-22T08:40:12.890546+0000","last_clean":"2024-04-22T08:40:12.890546+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:12.890546+0000","last_undegraded":"2024-04-22T08:40:12.890546+0000","last_fullsized":"2024-04-22T08:40:12.890546+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159391744,"data_stored":159381307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117155,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452520,"kb_used_data":96032,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93919320,"statfs":{"total":96636764160,"available":96173383680,"internally_reserved":0,"allocated":98336768,"data_stored":98029553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.59199999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519372,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852468,"statfs":{"total":96636764160,"available":96104927232,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313384,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97140736,"data_stored":97137257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:22.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:22.394 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:22.394 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:22.414 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-04-22T08:40:21.574566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1483888,"kb_used_data":157712,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281631632,"statfs":{"total":289910292480,"available":288390791168,"internally_reserved":0,"allocated":161497088,"data_stored":160581905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.092803"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20369,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215643+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:05.215643+0000","last_peered":"2024-04-22T08:40:05.215643+0000","last_clean":"2024-04-22T08:40:05.215643+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:05.215643+0000","last_undegraded":"2024-04-22T08:40:05.215643+0000","last_fullsized":"2024-04-22T08:40:05.215643+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216126+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:05.216126+0000","last_peered":"2024-04-22T08:40:05.216126+0000","last_clean":"2024-04-22T08:40:05.216126+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:05.216126+0000","last_undegraded":"2024-04-22T08:40:05.216126+0000","last_fullsized":"2024-04-22T08:40:05.216126+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215311+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:05.215311+0000","last_peered":"2024-04-22T08:40:05.215311+0000","last_clean":"2024-04-22T08:40:05.215311+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:05.215311+0000","last_undegraded":"2024-04-22T08:40:05.215311+0000","last_fullsized":"2024-04-22T08:40:05.215311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":26385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:17.789841+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:17.789841+0000","last_peered":"2024-04-22T08:40:17.789841+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:17.789841+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3052,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-722}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27612,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215574+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:05.215574+0000","last_peered":"2024-04-22T08:40:05.215574+0000","last_clean":"2024-04-22T08:40:05.215574+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:05.215574+0000","last_undegraded":"2024-04-22T08:40:05.215574+0000","last_fullsized":"2024-04-22T08:40:05.215574+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27648,"reported_epoch":55,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:12.890546+0000","last_change":"2024-04-22T08:40:07.906799+0000","last_active":"2024-04-22T08:40:12.890546+0000","last_peered":"2024-04-22T08:40:12.890546+0000","last_clean":"2024-04-22T08:40:12.890546+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:12.890546+0000","last_undegraded":"2024-04-22T08:40:12.890546+0000","last_fullsized":"2024-04-22T08:40:12.890546+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.215466+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:05.215466+0000","last_peered":"2024-04-22T08:40:05.215466+0000","last_clean":"2024-04-22T08:40:05.215466+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:05.215466+0000","last_undegraded":"2024-04-22T08:40:05.215466+0000","last_fullsized":"2024-04-22T08:40:05.215466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159391744,"data_stored":159381307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117155,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452520,"kb_used_data":96032,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93919320,"statfs":{"total":96636764160,"available":96173383680,"internally_reserved":0,"allocated":98336768,"data_stored":98029553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.59199999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519372,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852468,"statfs":{"total":96636764160,"available":96104927232,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313384,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97140736,"data_stored":97137257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:22.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:22.786 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:22.806 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-04-22T08:40:22.484061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1480300,"kb_used_data":154120,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281635220,"statfs":{"total":289910292480,"available":288394465280,"internally_reserved":0,"allocated":157818880,"data_stored":156898260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001632"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20370,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938824+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:06.938824+0000","last_peered":"2024-04-22T08:40:06.938824+0000","last_clean":"2024-04-22T08:40:06.938824+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:06.938824+0000","last_undegraded":"2024-04-22T08:40:06.938824+0000","last_fullsized":"2024-04-22T08:40:06.938824+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20167,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.939279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:06.939279+0000","last_peered":"2024-04-22T08:40:06.939279+0000","last_clean":"2024-04-22T08:40:06.939279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:06.939279+0000","last_undegraded":"2024-04-22T08:40:06.939279+0000","last_fullsized":"2024-04-22T08:40:06.939279+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938507+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:06.938507+0000","last_peered":"2024-04-22T08:40:06.938507+0000","last_clean":"2024-04-22T08:40:06.938507+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:06.938507+0000","last_undegraded":"2024-04-22T08:40:06.938507+0000","last_fullsized":"2024-04-22T08:40:06.938507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":26385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:17.789841+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:17.789841+0000","last_peered":"2024-04-22T08:40:17.789841+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:17.789841+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3052,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-722}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938392+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:06.938392+0000","last_peered":"2024-04-22T08:40:06.938392+0000","last_clean":"2024-04-22T08:40:06.938392+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:06.938392+0000","last_undegraded":"2024-04-22T08:40:06.938392+0000","last_fullsized":"2024-04-22T08:40:06.938392+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27651,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:20.574963+0000","last_change":"2024-04-22T08:40:20.894640+0000","last_active":"2024-04-22T08:40:20.574963+0000","last_peered":"2024-04-22T08:40:20.574963+0000","last_clean":"2024-04-22T08:40:20.574963+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.574963+0000","last_undegraded":"2024-04-22T08:40:20.574963+0000","last_fullsized":"2024-04-22T08:40:20.574963+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938912+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:06.938912+0000","last_peered":"2024-04-22T08:40:06.938912+0000","last_clean":"2024-04-22T08:40:06.938912+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:06.938912+0000","last_undegraded":"2024-04-22T08:40:06.938912+0000","last_fullsized":"2024-04-22T08:40:06.938912+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155705344,"data_stored":155694907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117156,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448932,"kb_used_data":92440,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93922908,"statfs":{"total":96636764160,"available":96177057792,"internally_reserved":0,"allocated":94658560,"data_stored":94345908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.58899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519372,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852468,"statfs":{"total":96636764160,"available":96104927232,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313384,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93450857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:22.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:22.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 20905, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:05.217436+0000', 'last_change': '2024-04-22T08:27:56.764089+0000', 'last_active': '2024-04-22T08:40:05.217436+0000', 'last_peered': '2024-04-22T08:40:05.217436+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:05.217436+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:22.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-04-22T08:40:22.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'20186", 'reported_seq': 26385, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:17.789841+0000', 'last_change': '2024-04-22T08:40:04.724914+0000', 'last_active': '2024-04-22T08:40:17.789841+0000', 'last_peered': '2024-04-22T08:40:17.789841+0000', 'last_clean': '2024-04-22T08:27:54.070394+0000', 'last_became_active': '2024-04-22T08:27:56.745750+0000', 'last_became_peered': '2024-04-22T08:27:56.745750+0000', 'last_unstale': '2024-04-22T08:40:17.789841+0000', 'last_undegraded': '2024-04-22T08:27:56.237303+0000', 'last_fullsized': '2024-04-22T08:27:56.085708+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 724, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20184, 'num_write_kb': 40360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3052, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -722}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:22.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:22.808 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27651, 'reported_epoch': 56, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:20.574963+0000', 'last_change': '2024-04-22T08:40:20.894640+0000', 'last_active': '2024-04-22T08:40:20.574963+0000', 'last_peered': '2024-04-22T08:40:20.574963+0000', 'last_clean': '2024-04-22T08:40:20.574963+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:20.574963+0000', 'last_undegraded': '2024-04-22T08:40:20.574963+0000', 'last_fullsized': '2024-04-22T08:40:20.574963+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:22.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:23.217 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:23.217 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4192/26 objects degraded (16123.077%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":858,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1515827200,"bytes_avail":288394465280,"bytes_total":289910292480,"degraded_objects":4192,"degraded_total":26,"degraded_ratio":161.23076923076923,"recovering_objects_per_sec":158,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:23.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:23.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:23.558 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:23.577 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-04-22T08:40:22.484061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1480300,"kb_used_data":154120,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281635220,"statfs":{"total":289910292480,"available":288394465280,"internally_reserved":0,"allocated":157818880,"data_stored":156898260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001632"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":20905,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:05.217436+0000","last_change":"2024-04-22T08:27:56.764089+0000","last_active":"2024-04-22T08:40:05.217436+0000","last_peered":"2024-04-22T08:40:05.217436+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:05.217436+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20370,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938824+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:06.938824+0000","last_peered":"2024-04-22T08:40:06.938824+0000","last_clean":"2024-04-22T08:40:06.938824+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:06.938824+0000","last_undegraded":"2024-04-22T08:40:06.938824+0000","last_fullsized":"2024-04-22T08:40:06.938824+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20421,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220121+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:05.220121+0000","last_peered":"2024-04-22T08:40:05.220121+0000","last_clean":"2024-04-22T08:40:05.220121+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:05.220121+0000","last_undegraded":"2024-04-22T08:40:05.220121+0000","last_fullsized":"2024-04-22T08:40:05.220121+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20167,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.939279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:06.939279+0000","last_peered":"2024-04-22T08:40:06.939279+0000","last_clean":"2024-04-22T08:40:06.939279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:06.939279+0000","last_undegraded":"2024-04-22T08:40:06.939279+0000","last_fullsized":"2024-04-22T08:40:06.939279+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20080,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.216550+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:06.216550+0000","last_peered":"2024-04-22T08:40:06.216550+0000","last_clean":"2024-04-22T08:40:06.216550+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:06.216550+0000","last_undegraded":"2024-04-22T08:40:06.216550+0000","last_fullsized":"2024-04-22T08:40:06.216550+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28005,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220755+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:05.220755+0000","last_peered":"2024-04-22T08:40:05.220755+0000","last_clean":"2024-04-22T08:40:05.220755+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:05.220755+0000","last_undegraded":"2024-04-22T08:40:05.220755+0000","last_fullsized":"2024-04-22T08:40:05.220755+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27622,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219953+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:05.219953+0000","last_peered":"2024-04-22T08:40:05.219953+0000","last_clean":"2024-04-22T08:40:05.219953+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:05.219953+0000","last_undegraded":"2024-04-22T08:40:05.219953+0000","last_fullsized":"2024-04-22T08:40:05.219953+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19973,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216633+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:05.216633+0000","last_peered":"2024-04-22T08:40:05.216633+0000","last_clean":"2024-04-22T08:40:05.216633+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:05.216633+0000","last_undegraded":"2024-04-22T08:40:05.216633+0000","last_fullsized":"2024-04-22T08:40:05.216633+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219946+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:05.219946+0000","last_peered":"2024-04-22T08:40:05.219946+0000","last_clean":"2024-04-22T08:40:05.219946+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:05.219946+0000","last_undegraded":"2024-04-22T08:40:05.219946+0000","last_fullsized":"2024-04-22T08:40:05.219946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20697,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220241+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:05.220241+0000","last_peered":"2024-04-22T08:40:05.220241+0000","last_clean":"2024-04-22T08:40:05.220241+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:05.220241+0000","last_undegraded":"2024-04-22T08:40:05.220241+0000","last_fullsized":"2024-04-22T08:40:05.220241+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938507+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:06.938507+0000","last_peered":"2024-04-22T08:40:06.938507+0000","last_clean":"2024-04-22T08:40:06.938507+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:06.938507+0000","last_undegraded":"2024-04-22T08:40:06.938507+0000","last_fullsized":"2024-04-22T08:40:06.938507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":26385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:17.789841+0000","last_change":"2024-04-22T08:40:04.724914+0000","last_active":"2024-04-22T08:40:17.789841+0000","last_peered":"2024-04-22T08:40:17.789841+0000","last_clean":"2024-04-22T08:27:54.070394+0000","last_became_active":"2024-04-22T08:27:56.745750+0000","last_became_peered":"2024-04-22T08:27:56.745750+0000","last_unstale":"2024-04-22T08:40:17.789841+0000","last_undegraded":"2024-04-22T08:27:56.237303+0000","last_fullsized":"2024-04-22T08:27:56.085708+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3052,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-722}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.216693+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:05.216693+0000","last_peered":"2024-04-22T08:40:05.216693+0000","last_clean":"2024-04-22T08:40:05.216693+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:05.216693+0000","last_undegraded":"2024-04-22T08:40:05.216693+0000","last_fullsized":"2024-04-22T08:40:05.216693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938392+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:06.938392+0000","last_peered":"2024-04-22T08:40:06.938392+0000","last_clean":"2024-04-22T08:40:06.938392+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:06.938392+0000","last_undegraded":"2024-04-22T08:40:06.938392+0000","last_fullsized":"2024-04-22T08:40:06.938392+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.219992+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:05.219992+0000","last_peered":"2024-04-22T08:40:05.219992+0000","last_clean":"2024-04-22T08:40:05.219992+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:05.219992+0000","last_undegraded":"2024-04-22T08:40:05.219992+0000","last_fullsized":"2024-04-22T08:40:05.219992+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27651,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:20.574963+0000","last_change":"2024-04-22T08:40:20.894640+0000","last_active":"2024-04-22T08:40:20.574963+0000","last_peered":"2024-04-22T08:40:20.574963+0000","last_clean":"2024-04-22T08:40:20.574963+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.574963+0000","last_undegraded":"2024-04-22T08:40:20.574963+0000","last_fullsized":"2024-04-22T08:40:20.574963+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20910,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220310+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:05.220310+0000","last_peered":"2024-04-22T08:40:05.220310+0000","last_clean":"2024-04-22T08:40:05.220310+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:05.220310+0000","last_undegraded":"2024-04-22T08:40:05.220310+0000","last_fullsized":"2024-04-22T08:40:05.220310+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220406+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:05.220406+0000","last_peered":"2024-04-22T08:40:05.220406+0000","last_clean":"2024-04-22T08:40:05.220406+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:05.220406+0000","last_undegraded":"2024-04-22T08:40:05.220406+0000","last_fullsized":"2024-04-22T08:40:05.220406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20438,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.333117+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:06.333117+0000","last_peered":"2024-04-22T08:40:06.333117+0000","last_clean":"2024-04-22T08:40:06.333117+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:06.333117+0000","last_undegraded":"2024-04-22T08:40:06.333117+0000","last_fullsized":"2024-04-22T08:40:06.333117+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938912+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:06.938912+0000","last_peered":"2024-04-22T08:40:06.938912+0000","last_clean":"2024-04-22T08:40:06.938912+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:06.938912+0000","last_undegraded":"2024-04-22T08:40:06.938912+0000","last_fullsized":"2024-04-22T08:40:06.938912+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20664,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220331+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:05.220331+0000","last_peered":"2024-04-22T08:40:05.220331+0000","last_clean":"2024-04-22T08:40:05.220331+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:05.220331+0000","last_undegraded":"2024-04-22T08:40:05.220331+0000","last_fullsized":"2024-04-22T08:40:05.220331+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220203+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:05.220203+0000","last_peered":"2024-04-22T08:40:05.220203+0000","last_clean":"2024-04-22T08:40:05.220203+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:05.220203+0000","last_undegraded":"2024-04-22T08:40:05.220203+0000","last_fullsized":"2024-04-22T08:40:05.220203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20737,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T08:40:05.220219+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:05.220219+0000","last_peered":"2024-04-22T08:40:05.220219+0000","last_clean":"2024-04-22T08:40:05.220219+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:05.220219+0000","last_undegraded":"2024-04-22T08:40:05.220219+0000","last_fullsized":"2024-04-22T08:40:05.220219+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41415,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155705344,"data_stored":155694907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117156,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448932,"kb_used_data":92440,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93922908,"statfs":{"total":96636764160,"available":96177057792,"internally_reserved":0,"allocated":94658560,"data_stored":94345908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.58899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519372,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852468,"statfs":{"total":96636764160,"available":96104927232,"internally_reserved":0,"allocated":462848,"data_stored":155633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.86599999999999999,"15min":1.2210000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":0.80000000000000004,"5min":7.9820000000000002,"15min":15.989000000000001},"last":0.316},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.94499999999999995,"15min":1.145},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":0.89000000000000001,"5min":8.0470000000000006,"15min":20.149000000000001},"last":0.45200000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313384,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511996,"kb_used_data":61228,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859844,"statfs":{"total":96636764160,"available":96112480256,"internally_reserved":0,"allocated":62697472,"data_stored":62396719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.374},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93450857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:26.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:26.896 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:26.896 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:26.914 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-22T08:40:26.485121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1488532,"kb_used_data":154152,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281626988,"statfs":{"total":289910292480,"available":288386035712,"internally_reserved":0,"allocated":157851648,"data_stored":156909238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001988"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":21657,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:22.790656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:22.790656+0000","last_peered":"2024-04-22T08:40:22.790656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:22.790656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3094}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20370,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938824+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:06.938824+0000","last_peered":"2024-04-22T08:40:06.938824+0000","last_clean":"2024-04-22T08:40:06.938824+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:06.938824+0000","last_undegraded":"2024-04-22T08:40:06.938824+0000","last_fullsized":"2024-04-22T08:40:06.938824+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20167,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.939279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:06.939279+0000","last_peered":"2024-04-22T08:40:06.939279+0000","last_clean":"2024-04-22T08:40:06.939279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:06.939279+0000","last_undegraded":"2024-04-22T08:40:06.939279+0000","last_fullsized":"2024-04-22T08:40:06.939279+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938507+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:06.938507+0000","last_peered":"2024-04-22T08:40:06.938507+0000","last_clean":"2024-04-22T08:40:06.938507+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:06.938507+0000","last_undegraded":"2024-04-22T08:40:06.938507+0000","last_fullsized":"2024-04-22T08:40:06.938507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938392+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:06.938392+0000","last_peered":"2024-04-22T08:40:06.938392+0000","last_clean":"2024-04-22T08:40:06.938392+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:06.938392+0000","last_undegraded":"2024-04-22T08:40:06.938392+0000","last_fullsized":"2024-04-22T08:40:06.938392+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559259+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:20.559259+0000","last_peered":"2024-04-22T08:40:20.559259+0000","last_clean":"2024-04-22T08:40:20.559259+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:20.559259+0000","last_undegraded":"2024-04-22T08:40:20.559259+0000","last_fullsized":"2024-04-22T08:40:20.559259+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27651,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:20.574963+0000","last_change":"2024-04-22T08:40:20.894640+0000","last_active":"2024-04-22T08:40:20.574963+0000","last_peered":"2024-04-22T08:40:20.574963+0000","last_clean":"2024-04-22T08:40:20.574963+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.574963+0000","last_undegraded":"2024-04-22T08:40:20.574963+0000","last_fullsized":"2024-04-22T08:40:20.574963+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558808+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:20.558808+0000","last_peered":"2024-04-22T08:40:20.558808+0000","last_clean":"2024-04-22T08:40:20.558808+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:20.558808+0000","last_undegraded":"2024-04-22T08:40:20.558808+0000","last_fullsized":"2024-04-22T08:40:20.558808+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938912+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:06.938912+0000","last_peered":"2024-04-22T08:40:06.938912+0000","last_clean":"2024-04-22T08:40:06.938912+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:06.938912+0000","last_undegraded":"2024-04-22T08:40:06.938912+0000","last_fullsized":"2024-04-22T08:40:06.938912+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155705344,"data_stored":155694907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117156,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448932,"kb_used_data":92440,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93922908,"statfs":{"total":96636764160,"available":96177057792,"internally_reserved":0,"allocated":94658560,"data_stored":94345908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.58899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527588,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844252,"statfs":{"total":96636764160,"available":96096514048,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313385,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.192},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.216}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93450857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:26.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:27.228 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:27.228 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:27.246 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-22T08:40:26.485121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1488532,"kb_used_data":154152,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281626988,"statfs":{"total":289910292480,"available":288386035712,"internally_reserved":0,"allocated":157851648,"data_stored":156909238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001988"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":21657,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:22.790656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:22.790656+0000","last_peered":"2024-04-22T08:40:22.790656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:22.790656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3094}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20370,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938824+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:06.938824+0000","last_peered":"2024-04-22T08:40:06.938824+0000","last_clean":"2024-04-22T08:40:06.938824+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:06.938824+0000","last_undegraded":"2024-04-22T08:40:06.938824+0000","last_fullsized":"2024-04-22T08:40:06.938824+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20167,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.939279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:06.939279+0000","last_peered":"2024-04-22T08:40:06.939279+0000","last_clean":"2024-04-22T08:40:06.939279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:06.939279+0000","last_undegraded":"2024-04-22T08:40:06.939279+0000","last_fullsized":"2024-04-22T08:40:06.939279+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938507+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:06.938507+0000","last_peered":"2024-04-22T08:40:06.938507+0000","last_clean":"2024-04-22T08:40:06.938507+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:06.938507+0000","last_undegraded":"2024-04-22T08:40:06.938507+0000","last_fullsized":"2024-04-22T08:40:06.938507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938392+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:06.938392+0000","last_peered":"2024-04-22T08:40:06.938392+0000","last_clean":"2024-04-22T08:40:06.938392+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:06.938392+0000","last_undegraded":"2024-04-22T08:40:06.938392+0000","last_fullsized":"2024-04-22T08:40:06.938392+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559259+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:20.559259+0000","last_peered":"2024-04-22T08:40:20.559259+0000","last_clean":"2024-04-22T08:40:20.559259+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:20.559259+0000","last_undegraded":"2024-04-22T08:40:20.559259+0000","last_fullsized":"2024-04-22T08:40:20.559259+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27651,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:20.574963+0000","last_change":"2024-04-22T08:40:20.894640+0000","last_active":"2024-04-22T08:40:20.574963+0000","last_peered":"2024-04-22T08:40:20.574963+0000","last_clean":"2024-04-22T08:40:20.574963+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.574963+0000","last_undegraded":"2024-04-22T08:40:20.574963+0000","last_fullsized":"2024-04-22T08:40:20.574963+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558808+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:20.558808+0000","last_peered":"2024-04-22T08:40:20.558808+0000","last_clean":"2024-04-22T08:40:20.558808+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:20.558808+0000","last_undegraded":"2024-04-22T08:40:20.558808+0000","last_fullsized":"2024-04-22T08:40:20.558808+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938912+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:06.938912+0000","last_peered":"2024-04-22T08:40:06.938912+0000","last_clean":"2024-04-22T08:40:06.938912+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:06.938912+0000","last_undegraded":"2024-04-22T08:40:06.938912+0000","last_fullsized":"2024-04-22T08:40:06.938912+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155705344,"data_stored":155694907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117156,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448932,"kb_used_data":92440,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93922908,"statfs":{"total":96636764160,"available":96177057792,"internally_reserved":0,"allocated":94658560,"data_stored":94345908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.58899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527588,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844252,"statfs":{"total":96636764160,"available":96096514048,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313385,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.192},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.216}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93450857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:27.248 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:27.248 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 21657, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:22.790656+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:22.790656+0000', 'last_peered': '2024-04-22T08:40:22.790656+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:22.790656+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 700, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3094}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:27.248 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:27.248 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27651, 'reported_epoch': 56, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:20.574963+0000', 'last_change': '2024-04-22T08:40:20.894640+0000', 'last_active': '2024-04-22T08:40:20.574963+0000', 'last_peered': '2024-04-22T08:40:20.574963+0000', 'last_clean': '2024-04-22T08:40:20.574963+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:20.574963+0000', 'last_undegraded': '2024-04-22T08:40:20.574963+0000', 'last_fullsized': '2024-04-22T08:40:20.574963+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:27.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:27.629 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:27.629 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3094/26 objects degraded (11900.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":862,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1524256768,"bytes_avail":288386035712,"bytes_total":289910292480,"degraded_objects":3094,"degraded_total":26,"degraded_ratio":119,"recovering_objects_per_sec":137,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:27.649 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:27.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:27.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:27.951 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:27.970 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-22T08:40:26.485121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1488532,"kb_used_data":154152,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281626988,"statfs":{"total":289910292480,"available":288386035712,"internally_reserved":0,"allocated":157851648,"data_stored":156909238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001988"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":21657,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:22.790656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:22.790656+0000","last_peered":"2024-04-22T08:40:22.790656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:22.790656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3094}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20370,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938824+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:06.938824+0000","last_peered":"2024-04-22T08:40:06.938824+0000","last_clean":"2024-04-22T08:40:06.938824+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:06.938824+0000","last_undegraded":"2024-04-22T08:40:06.938824+0000","last_fullsized":"2024-04-22T08:40:06.938824+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20167,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.939279+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:06.939279+0000","last_peered":"2024-04-22T08:40:06.939279+0000","last_clean":"2024-04-22T08:40:06.939279+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:06.939279+0000","last_undegraded":"2024-04-22T08:40:06.939279+0000","last_fullsized":"2024-04-22T08:40:06.939279+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938507+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:06.938507+0000","last_peered":"2024-04-22T08:40:06.938507+0000","last_clean":"2024-04-22T08:40:06.938507+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:06.938507+0000","last_undegraded":"2024-04-22T08:40:06.938507+0000","last_fullsized":"2024-04-22T08:40:06.938507+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938392+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:06.938392+0000","last_peered":"2024-04-22T08:40:06.938392+0000","last_clean":"2024-04-22T08:40:06.938392+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:06.938392+0000","last_undegraded":"2024-04-22T08:40:06.938392+0000","last_fullsized":"2024-04-22T08:40:06.938392+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559259+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:20.559259+0000","last_peered":"2024-04-22T08:40:20.559259+0000","last_clean":"2024-04-22T08:40:20.559259+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:20.559259+0000","last_undegraded":"2024-04-22T08:40:20.559259+0000","last_fullsized":"2024-04-22T08:40:20.559259+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27651,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:20.574963+0000","last_change":"2024-04-22T08:40:20.894640+0000","last_active":"2024-04-22T08:40:20.574963+0000","last_peered":"2024-04-22T08:40:20.574963+0000","last_clean":"2024-04-22T08:40:20.574963+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.574963+0000","last_undegraded":"2024-04-22T08:40:20.574963+0000","last_fullsized":"2024-04-22T08:40:20.574963+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558808+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:20.558808+0000","last_peered":"2024-04-22T08:40:20.558808+0000","last_clean":"2024-04-22T08:40:20.558808+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:20.558808+0000","last_undegraded":"2024-04-22T08:40:20.558808+0000","last_fullsized":"2024-04-22T08:40:20.558808+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-22T08:40:06.938912+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:06.938912+0000","last_peered":"2024-04-22T08:40:06.938912+0000","last_clean":"2024-04-22T08:40:06.938912+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:06.938912+0000","last_undegraded":"2024-04-22T08:40:06.938912+0000","last_fullsized":"2024-04-22T08:40:06.938912+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42513,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155705344,"data_stored":155694907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117156,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448932,"kb_used_data":92440,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93922908,"statfs":{"total":96636764160,"available":96177057792,"internally_reserved":0,"allocated":94658560,"data_stored":94345908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.88300000000000001,"15min":0.97999999999999998},"min":{"1min":0.21199999999999999,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.74299999999999999,"5min":6.3860000000000001,"15min":13.635},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.82799999999999996,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.875,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.58899999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527588,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844252,"statfs":{"total":96636764160,"available":96096514048,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.95099999999999996,"15min":0.76400000000000001},"min":{"1min":0.27900000000000003,"5min":0.19500000000000001,"15min":0.17000000000000001},"max":{"1min":0.97799999999999998,"5min":9.0660000000000007,"15min":9.0660000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.0149999999999999,"15min":0.91300000000000003},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.0469999999999999,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":21,"seq":90194313385,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.192},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.216}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93450857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:30.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:31.313 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:31.313 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:31.331 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-04-22T08:40:30.486354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472164,"kb_used_data":150084,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281643356,"statfs":{"total":289910292480,"available":288402796544,"internally_reserved":0,"allocated":153686016,"data_stored":152738148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.929127"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":22379,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:27.790265+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:27.790265+0000","last_peered":"2024-04-22T08:40:27.790265+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:27.790265+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27652,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.894640+0000","last_change":"2024-04-22T08:40:21.870102+0000","last_active":"2024-04-22T08:40:20.894640+0000","last_peered":"2024-04-22T08:40:20.894640+0000","last_clean":"2024-04-22T08:40:20.894640+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.894640+0000","last_undegraded":"2024-04-22T08:40:20.894640+0000","last_fullsized":"2024-04-22T08:40:20.894640+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":151408640,"data_stored":151398203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444860,"kb_used_data":88372,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93926980,"statfs":{"total":96636764160,"available":96181227520,"internally_reserved":0,"allocated":90492928,"data_stored":90174818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.69899999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.65900000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856548,"statfs":{"total":96636764160,"available":96109105152,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.622}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.54200000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313386,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.751}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":1.014},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":1.2110000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89157632,"data_stored":89154153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:31.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:31.640 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:31.641 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:31.658 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-04-22T08:40:30.486354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472164,"kb_used_data":150084,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281643356,"statfs":{"total":289910292480,"available":288402796544,"internally_reserved":0,"allocated":153686016,"data_stored":152738148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.929127"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":22379,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:27.790265+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:27.790265+0000","last_peered":"2024-04-22T08:40:27.790265+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:27.790265+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27652,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.894640+0000","last_change":"2024-04-22T08:40:21.870102+0000","last_active":"2024-04-22T08:40:20.894640+0000","last_peered":"2024-04-22T08:40:20.894640+0000","last_clean":"2024-04-22T08:40:20.894640+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.894640+0000","last_undegraded":"2024-04-22T08:40:20.894640+0000","last_fullsized":"2024-04-22T08:40:20.894640+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":151408640,"data_stored":151398203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444860,"kb_used_data":88372,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93926980,"statfs":{"total":96636764160,"available":96181227520,"internally_reserved":0,"allocated":90492928,"data_stored":90174818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.69899999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.65900000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856548,"statfs":{"total":96636764160,"available":96109105152,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.622}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.54200000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313386,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.751}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":1.014},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":1.2110000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89157632,"data_stored":89154153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:31.659 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:31.659 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 22379, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:27.790265+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:27.790265+0000', 'last_peered': '2024-04-22T08:40:27.790265+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:27.790265+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2733, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1061, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2733}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:31.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:32.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:32.047 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2733/26 objects degraded (10511.538%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":866,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1507495936,"bytes_avail":288402796544,"bytes_total":289910292480,"degraded_objects":2733,"degraded_total":26,"degraded_ratio":105.11538461538461,"recovering_objects_per_sec":146,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1459,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:32.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:32.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:32.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:32.392 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:32.410 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-04-22T08:40:30.486354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472164,"kb_used_data":150084,"kb_used_omap":12,"kb_used_meta":1322035,"kb_avail":281643356,"statfs":{"total":289910292480,"available":288402796544,"internally_reserved":0,"allocated":153686016,"data_stored":152738148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1353763950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.929127"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":22379,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:27.790265+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:27.790265+0000","last_peered":"2024-04-22T08:40:27.790265+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:27.790265+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27652,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.894640+0000","last_change":"2024-04-22T08:40:21.870102+0000","last_active":"2024-04-22T08:40:20.894640+0000","last_peered":"2024-04-22T08:40:20.894640+0000","last_clean":"2024-04-22T08:40:20.894640+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:20.894640+0000","last_undegraded":"2024-04-22T08:40:20.894640+0000","last_fullsized":"2024-04-22T08:40:20.894640+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42874,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":151408640,"data_stored":151398203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444860,"kb_used_data":88372,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93926980,"statfs":{"total":96636764160,"available":96181227520,"internally_reserved":0,"allocated":90492928,"data_stored":90174818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.69899999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.65900000000000003}]}]},{"osd":1,"up_from":33,"seq":141733920919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856548,"statfs":{"total":96636764160,"available":96109105152,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.622}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.54200000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313386,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.751}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":1.014},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":1.2110000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89157632,"data_stored":89154153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:35.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:35.765 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:35.765 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:35.784 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-04-22T08:40:34.487392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472060,"kb_used_data":145884,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281643460,"statfs":{"total":289910292480,"available":288402903040,"internally_reserved":0,"allocated":149385216,"data_stored":148437348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23039,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:32.784758+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:32.784758+0000","last_peered":"2024-04-22T08:40:32.784758+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:32.784758+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27654,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:27.905642+0000","last_change":"2024-04-22T08:40:27.905642+0000","last_active":"2024-04-22T08:40:27.905642+0000","last_peered":"2024-04-22T08:40:27.905642+0000","last_clean":"2024-04-22T08:40:27.905642+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:27.905642+0000","last_undegraded":"2024-04-22T08:40:27.905642+0000","last_fullsized":"2024-04-22T08:40:27.905642+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147107840,"data_stored":147097403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117158,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440660,"kb_used_data":84172,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93931180,"statfs":{"total":96636764160,"available":96185528320,"internally_reserved":0,"allocated":86192128,"data_stored":85874018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68799999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313387,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84856832,"data_stored":84853353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:35.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:36.117 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:36.117 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:36.137 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-04-22T08:40:34.487392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472060,"kb_used_data":145884,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281643460,"statfs":{"total":289910292480,"available":288402903040,"internally_reserved":0,"allocated":149385216,"data_stored":148437348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23039,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:32.784758+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:32.784758+0000","last_peered":"2024-04-22T08:40:32.784758+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:32.784758+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27654,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:27.905642+0000","last_change":"2024-04-22T08:40:27.905642+0000","last_active":"2024-04-22T08:40:27.905642+0000","last_peered":"2024-04-22T08:40:27.905642+0000","last_clean":"2024-04-22T08:40:27.905642+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:27.905642+0000","last_undegraded":"2024-04-22T08:40:27.905642+0000","last_fullsized":"2024-04-22T08:40:27.905642+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147107840,"data_stored":147097403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117158,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440660,"kb_used_data":84172,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93931180,"statfs":{"total":96636764160,"available":96185528320,"internally_reserved":0,"allocated":86192128,"data_stored":85874018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68799999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313387,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84856832,"data_stored":84853353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:36.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:36.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 23039, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:32.784758+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:32.784758+0000', 'last_peered': '2024-04-22T08:40:32.784758+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:32.784758+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2403, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1391, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2403}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:36.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:36.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27654, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:27.905642+0000', 'last_change': '2024-04-22T08:40:27.905642+0000', 'last_active': '2024-04-22T08:40:27.905642+0000', 'last_peered': '2024-04-22T08:40:27.905642+0000', 'last_clean': '2024-04-22T08:40:27.905642+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:27.905642+0000', 'last_undegraded': '2024-04-22T08:40:27.905642+0000', 'last_fullsized': '2024-04-22T08:40:27.905642+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:36.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:36.504 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2403/26 objects degraded (9242.308%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":871,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1507389440,"bytes_avail":288402903040,"bytes_total":289910292480,"degraded_objects":2403,"degraded_total":26,"degraded_ratio":92.42307692307692,"recovering_objects_per_sec":149,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1789,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:36.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:36.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:36.831 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:36.849 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-04-22T08:40:36.488172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472060,"kb_used_data":145884,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281643460,"statfs":{"total":289910292480,"available":288402903040,"internally_reserved":0,"allocated":149385216,"data_stored":148437348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003712"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23039,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:32.784758+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:32.784758+0000","last_peered":"2024-04-22T08:40:32.784758+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:32.784758+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20423,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558952+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:20.558952+0000","last_peered":"2024-04-22T08:40:20.558952+0000","last_clean":"2024-04-22T08:40:20.558952+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:20.558952+0000","last_undegraded":"2024-04-22T08:40:20.558952+0000","last_fullsized":"2024-04-22T08:40:20.558952+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575254+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:20.575254+0000","last_peered":"2024-04-22T08:40:20.575254+0000","last_clean":"2024-04-22T08:40:20.575254+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:20.575254+0000","last_undegraded":"2024-04-22T08:40:20.575254+0000","last_fullsized":"2024-04-22T08:40:20.575254+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27654,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:27.905642+0000","last_change":"2024-04-22T08:40:27.905642+0000","last_active":"2024-04-22T08:40:27.905642+0000","last_peered":"2024-04-22T08:40:27.905642+0000","last_clean":"2024-04-22T08:40:27.905642+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:27.905642+0000","last_undegraded":"2024-04-22T08:40:27.905642+0000","last_fullsized":"2024-04-22T08:40:27.905642+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558728+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:20.558728+0000","last_peered":"2024-04-22T08:40:20.558728+0000","last_clean":"2024-04-22T08:40:20.558728+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:20.558728+0000","last_undegraded":"2024-04-22T08:40:20.558728+0000","last_fullsized":"2024-04-22T08:40:20.558728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43204,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147107840,"data_stored":147097403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117158,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440660,"kb_used_data":84172,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93931180,"statfs":{"total":96636764160,"available":96185528320,"internally_reserved":0,"allocated":86192128,"data_stored":85874018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68799999999999994}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313387,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.92700000000000005,"15min":0.95799999999999996},"min":{"1min":0.25600000000000001,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":0.871,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.3879999999999999,"15min":1.069},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.94299999999999995,"5min":11.141999999999999,"15min":12.131},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84856832,"data_stored":84853353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:39.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:40.198 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:40.198 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:40.218 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-04-22T08:40:38.488574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471956,"kb_used_data":141684,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281643564,"statfs":{"total":289910292480,"available":288403009536,"internally_reserved":0,"allocated":145084416,"data_stored":144136548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23761,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:37.784430+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:37.784430+0000","last_peered":"2024-04-22T08:40:37.784430+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:37.784430+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2042}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27657,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:32.843387+0000","last_change":"2024-04-22T08:40:32.843387+0000","last_active":"2024-04-22T08:40:32.843387+0000","last_peered":"2024-04-22T08:40:32.843387+0000","last_clean":"2024-04-22T08:40:32.843387+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:32.843387+0000","last_undegraded":"2024-04-22T08:40:32.843387+0000","last_fullsized":"2024-04-22T08:40:32.843387+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":142807040,"data_stored":142796603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117159,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440556,"kb_used_data":79972,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93931284,"statfs":{"total":96636764160,"available":96185634816,"internally_reserved":0,"allocated":81891328,"data_stored":81573218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.5},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.625}]}]},{"osd":1,"up_from":33,"seq":141733920921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.62}]}]},{"osd":0,"up_from":21,"seq":90194313388,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80556032,"data_stored":80552553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:40.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:40.546 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:40.564 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-04-22T08:40:40.489217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471956,"kb_used_data":141684,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281643564,"statfs":{"total":289910292480,"available":288403009536,"internally_reserved":0,"allocated":145084416,"data_stored":144136548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003536"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23761,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:37.784430+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:37.784430+0000","last_peered":"2024-04-22T08:40:37.784430+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:37.784430+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2042}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27657,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:32.843387+0000","last_change":"2024-04-22T08:40:32.843387+0000","last_active":"2024-04-22T08:40:32.843387+0000","last_peered":"2024-04-22T08:40:32.843387+0000","last_clean":"2024-04-22T08:40:32.843387+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:32.843387+0000","last_undegraded":"2024-04-22T08:40:32.843387+0000","last_fullsized":"2024-04-22T08:40:32.843387+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":142807040,"data_stored":142796603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117159,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440556,"kb_used_data":79972,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93931284,"statfs":{"total":96636764160,"available":96185634816,"internally_reserved":0,"allocated":81891328,"data_stored":81573218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.5},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.625}]}]},{"osd":1,"up_from":33,"seq":141733920921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.62}]}]},{"osd":0,"up_from":21,"seq":90194313388,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80556032,"data_stored":80552553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:40.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:40.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 23761, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:37.784430+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:37.784430+0000', 'last_peered': '2024-04-22T08:40:37.784430+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:37.784430+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1752, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2042}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:40.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:40.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27657, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:32.843387+0000', 'last_change': '2024-04-22T08:40:32.843387+0000', 'last_active': '2024-04-22T08:40:32.843387+0000', 'last_peered': '2024-04-22T08:40:32.843387+0000', 'last_clean': '2024-04-22T08:40:32.843387+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:32.843387+0000', 'last_undegraded': '2024-04-22T08:40:32.843387+0000', 'last_fullsized': '2024-04-22T08:40:32.843387+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:40.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:40.937 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2042/26 objects degraded (7853.846%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":875,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1507282944,"bytes_avail":288403009536,"bytes_total":289910292480,"degraded_objects":2042,"degraded_total":26,"degraded_ratio":78.538461538461533,"recovering_objects_per_sec":57,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-22T08:38:42.456326+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:40.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:40.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:41.301 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:41.318 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-04-22T08:40:40.489217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471956,"kb_used_data":141684,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281643564,"statfs":{"total":289910292480,"available":288403009536,"internally_reserved":0,"allocated":145084416,"data_stored":144136548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003536"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":23761,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:37.784430+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:37.784430+0000","last_peered":"2024-04-22T08:40:37.784430+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:37.784430+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2042}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27657,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:32.843387+0000","last_change":"2024-04-22T08:40:32.843387+0000","last_active":"2024-04-22T08:40:32.843387+0000","last_peered":"2024-04-22T08:40:32.843387+0000","last_clean":"2024-04-22T08:40:32.843387+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:32.843387+0000","last_undegraded":"2024-04-22T08:40:32.843387+0000","last_fullsized":"2024-04-22T08:40:32.843387+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43565,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":142807040,"data_stored":142796603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117159,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440556,"kb_used_data":79972,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93931284,"statfs":{"total":96636764160,"available":96185634816,"internally_reserved":0,"allocated":81891328,"data_stored":81573218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.5},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.625}]}]},{"osd":1,"up_from":33,"seq":141733920921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.62}]}]},{"osd":0,"up_from":21,"seq":90194313388,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80556032,"data_stored":80552553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:44.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:44.644 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:44.664 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-22T08:40:44.490173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471852,"kb_used_data":137484,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281643668,"statfs":{"total":289910292480,"available":288403116032,"internally_reserved":0,"allocated":140783616,"data_stored":139835748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":24509,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:42.792656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:42.792656+0000","last_peered":"2024-04-22T08:40:42.792656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:42.792656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1668}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27659,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:39.889796+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:39.889796+0000","last_peered":"2024-04-22T08:40:39.889796+0000","last_clean":"2024-04-22T08:40:39.889796+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:39.889796+0000","last_undegraded":"2024-04-22T08:40:39.889796+0000","last_fullsized":"2024-04-22T08:40:39.889796+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138506240,"data_stored":138495803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117160,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436356,"kb_used_data":75772,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93935484,"statfs":{"total":96636764160,"available":96189935616,"internally_reserved":0,"allocated":77590528,"data_stored":77272418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.629},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.72799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523484,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848356,"statfs":{"total":96636764160,"available":96100716544,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313389,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76255232,"data_stored":76251753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:44.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:44.969 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:44.970 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:44.988 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-22T08:40:44.490173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471852,"kb_used_data":137484,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281643668,"statfs":{"total":289910292480,"available":288403116032,"internally_reserved":0,"allocated":140783616,"data_stored":139835748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":24509,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:42.792656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:42.792656+0000","last_peered":"2024-04-22T08:40:42.792656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:42.792656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1668}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27659,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:39.889796+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:39.889796+0000","last_peered":"2024-04-22T08:40:39.889796+0000","last_clean":"2024-04-22T08:40:39.889796+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:39.889796+0000","last_undegraded":"2024-04-22T08:40:39.889796+0000","last_fullsized":"2024-04-22T08:40:39.889796+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138506240,"data_stored":138495803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117160,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436356,"kb_used_data":75772,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93935484,"statfs":{"total":96636764160,"available":96189935616,"internally_reserved":0,"allocated":77590528,"data_stored":77272418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.629},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.72799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523484,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848356,"statfs":{"total":96636764160,"available":96100716544,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313389,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76255232,"data_stored":76251753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:44.990 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:44.990 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 24509, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:42.792656+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:42.792656+0000', 'last_peered': '2024-04-22T08:40:42.792656+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:42.792656+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1668, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2126, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1668}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:44.990 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:44.990 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27659, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:39.889796+0000', 'last_change': '2024-04-22T08:40:40.903517+0000', 'last_active': '2024-04-22T08:40:39.889796+0000', 'last_peered': '2024-04-22T08:40:39.889796+0000', 'last_clean': '2024-04-22T08:40:39.889796+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:39.889796+0000', 'last_undegraded': '2024-04-22T08:40:39.889796+0000', 'last_fullsized': '2024-04-22T08:40:39.889796+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:44.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:45.377 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1668/26 objects degraded (6415.385%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":880,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1507176448,"bytes_avail":288403116032,"bytes_total":289910292480,"degraded_objects":1668,"degraded_total":26,"degraded_ratio":64.15384615384616,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-22T08:40:42.489689+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:45.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:45.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:45.733 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:45.734 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:45.752 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-22T08:40:44.490173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471852,"kb_used_data":137484,"kb_used_omap":12,"kb_used_meta":1334323,"kb_avail":281643668,"statfs":{"total":289910292480,"available":288403116032,"internally_reserved":0,"allocated":140783616,"data_stored":139835748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1366346862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":24509,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:42.792656+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:42.792656+0000","last_peered":"2024-04-22T08:40:42.792656+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:42.792656+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1668}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27659,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:39.889796+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:39.889796+0000","last_peered":"2024-04-22T08:40:39.889796+0000","last_clean":"2024-04-22T08:40:39.889796+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:39.889796+0000","last_undegraded":"2024-04-22T08:40:39.889796+0000","last_fullsized":"2024-04-22T08:40:39.889796+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43939,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138506240,"data_stored":138495803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117160,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436356,"kb_used_data":75772,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93935484,"statfs":{"total":96636764160,"available":96189935616,"internally_reserved":0,"allocated":77590528,"data_stored":77272418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.629},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.72799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523484,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848356,"statfs":{"total":96636764160,"available":96100716544,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313389,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76255232,"data_stored":76251753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:48.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:49.398 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:49.398 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:49.415 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-22T08:40:48.491148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471748,"kb_used_data":133284,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281643772,"statfs":{"total":289910292480,"available":288403222528,"internally_reserved":0,"allocated":136482816,"data_stored":135534948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":25423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:47.797363+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:47.797363+0000","last_peered":"2024-04-22T08:40:47.797363+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:47.797363+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27662,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:42.897397+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:42.897397+0000","last_peered":"2024-04-22T08:40:42.897397+0000","last_clean":"2024-04-22T08:40:42.897397+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:42.897397+0000","last_undegraded":"2024-04-22T08:40:42.897397+0000","last_fullsized":"2024-04-22T08:40:42.897397+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134205440,"data_stored":134195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117161,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432156,"kb_used_data":71572,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93939684,"statfs":{"total":96636764160,"available":96194236416,"internally_reserved":0,"allocated":73289728,"data_stored":72971618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.307},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.34799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527580,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844260,"statfs":{"total":96636764160,"available":96096522240,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313390,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71954432,"data_stored":71950953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:49.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:49.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:49.726 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:49.744 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-22T08:40:48.491148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471748,"kb_used_data":133284,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281643772,"statfs":{"total":289910292480,"available":288403222528,"internally_reserved":0,"allocated":136482816,"data_stored":135534948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":25423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:47.797363+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:47.797363+0000","last_peered":"2024-04-22T08:40:47.797363+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:47.797363+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27662,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:42.897397+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:42.897397+0000","last_peered":"2024-04-22T08:40:42.897397+0000","last_clean":"2024-04-22T08:40:42.897397+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:42.897397+0000","last_undegraded":"2024-04-22T08:40:42.897397+0000","last_fullsized":"2024-04-22T08:40:42.897397+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134205440,"data_stored":134195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117161,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432156,"kb_used_data":71572,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93939684,"statfs":{"total":96636764160,"available":96194236416,"internally_reserved":0,"allocated":73289728,"data_stored":72971618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.307},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.34799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527580,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844260,"statfs":{"total":96636764160,"available":96096522240,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313390,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71954432,"data_stored":71950953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:49.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:49.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 25423, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:47.797363+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:47.797363+0000', 'last_peered': '2024-04-22T08:40:47.797363+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:47.797363+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2583, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1211}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:49.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:49.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27662, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:42.897397+0000', 'last_change': '2024-04-22T08:40:40.903517+0000', 'last_active': '2024-04-22T08:40:42.897397+0000', 'last_peered': '2024-04-22T08:40:42.897397+0000', 'last_clean': '2024-04-22T08:40:42.897397+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:42.897397+0000', 'last_undegraded': '2024-04-22T08:40:42.897397+0000', 'last_fullsized': '2024-04-22T08:40:42.897397+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:49.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:50.134 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:50.134 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1211/26 objects degraded (4657.692%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":885,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1507069952,"bytes_avail":288403222528,"bytes_total":289910292480,"degraded_objects":1211,"degraded_total":26,"degraded_ratio":46.576923076923073,"recovering_objects_per_sec":99,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-22T08:40:42.489689+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:50.152 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:50.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:50.446 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:50.447 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:50.465 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-22T08:40:48.491148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471748,"kb_used_data":133284,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281643772,"statfs":{"total":289910292480,"available":288403222528,"internally_reserved":0,"allocated":136482816,"data_stored":135534948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":25423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:47.797363+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:47.797363+0000","last_peered":"2024-04-22T08:40:47.797363+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:47.797363+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27662,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:42.897397+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:42.897397+0000","last_peered":"2024-04-22T08:40:42.897397+0000","last_clean":"2024-04-22T08:40:42.897397+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:42.897397+0000","last_undegraded":"2024-04-22T08:40:42.897397+0000","last_fullsized":"2024-04-22T08:40:42.897397+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134205440,"data_stored":134195003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117161,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432156,"kb_used_data":71572,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93939684,"statfs":{"total":96636764160,"available":96194236416,"internally_reserved":0,"allocated":73289728,"data_stored":72971618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.376}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.307},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.34799999999999998}]}]},{"osd":1,"up_from":33,"seq":141733920923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527580,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844260,"statfs":{"total":96636764160,"available":96096522240,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313390,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71954432,"data_stored":71950953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:53.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:53.796 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:53.796 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:53.815 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-04-22T08:40:52.492113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1467548,"kb_used_data":129084,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281647972,"statfs":{"total":289910292480,"available":288407523328,"internally_reserved":0,"allocated":132182016,"data_stored":131237627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002895"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":25423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:47.797363+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:47.797363+0000","last_peered":"2024-04-22T08:40:47.797363+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:47.797363+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27663,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:43.896227+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:43.896227+0000","last_peered":"2024-04-22T08:40:43.896227+0000","last_clean":"2024-04-22T08:40:43.896227+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:43.896227+0000","last_undegraded":"2024-04-22T08:40:43.896227+0000","last_fullsized":"2024-04-22T08:40:43.896227+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129904640,"data_stored":129897682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117162,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427956,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93943884,"statfs":{"total":96636764160,"available":96198537216,"internally_reserved":0,"allocated":68988928,"data_stored":68674297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.59499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527580,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844260,"statfs":{"total":96636764160,"available":96096522240,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313390,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67653632,"data_stored":67653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:53.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:54.122 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:54.122 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:54.140 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-04-22T08:40:52.492113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1467548,"kb_used_data":129084,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281647972,"statfs":{"total":289910292480,"available":288407523328,"internally_reserved":0,"allocated":132182016,"data_stored":131237627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002895"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":25423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:47.797363+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:47.797363+0000","last_peered":"2024-04-22T08:40:47.797363+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:47.797363+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27663,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:43.896227+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:43.896227+0000","last_peered":"2024-04-22T08:40:43.896227+0000","last_clean":"2024-04-22T08:40:43.896227+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:43.896227+0000","last_undegraded":"2024-04-22T08:40:43.896227+0000","last_fullsized":"2024-04-22T08:40:43.896227+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44396,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129904640,"data_stored":129897682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117162,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427956,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93943884,"statfs":{"total":96636764160,"available":96198537216,"internally_reserved":0,"allocated":68988928,"data_stored":68674297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.59499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527580,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":527099,"kb_avail":93844260,"statfs":{"total":96636764160,"available":96096522240,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":539750092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.497},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313390,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67653632,"data_stored":67653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:54.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:54.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 25423, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-22T08:40:47.797363+0000', 'last_change': '2024-04-22T08:40:20.014122+0000', 'last_active': '2024-04-22T08:40:47.797363+0000', 'last_peered': '2024-04-22T08:40:47.797363+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:47.797363+0000', 'last_undegraded': '2024-04-22T08:27:56.219112+0000', 'last_fullsized': '2024-04-22T08:27:56.098408+0000', 'mapping_epoch': 34, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2583, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1211}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:54.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:54.142 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27663, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:43.896227+0000', 'last_change': '2024-04-22T08:40:40.903517+0000', 'last_active': '2024-04-22T08:40:43.896227+0000', 'last_peered': '2024-04-22T08:40:43.896227+0000', 'last_clean': '2024-04-22T08:40:43.896227+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:43.896227+0000', 'last_undegraded': '2024-04-22T08:40:43.896227+0000', 'last_fullsized': '2024-04-22T08:40:43.896227+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:54.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:54.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:54.508 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1211/26 objects degraded (4657.692%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":889,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1502769152,"bytes_avail":288407523328,"bytes_total":289910292480,"degraded_objects":1211,"degraded_total":26,"degraded_ratio":46.576923076923073,"recovering_objects_per_sec":69,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":831,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-22T08:40:42.489689+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:54.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:54.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:54.833 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:54.834 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:54.852 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-04-22T08:40:54.492717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44896,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1455260,"kb_used_data":129084,"kb_used_omap":12,"kb_used_meta":1326131,"kb_avail":281660260,"statfs":{"total":289910292480,"available":288420106240,"internally_reserved":0,"allocated":132182016,"data_stored":131237627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1357958254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003129"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":26423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:52.800945+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:52.800945+0000","last_peered":"2024-04-22T08:40:52.800945+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:52.800945+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3083,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-711}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27663,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:43.896227+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:43.896227+0000","last_peered":"2024-04-22T08:40:43.896227+0000","last_clean":"2024-04-22T08:40:43.896227+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:43.896227+0000","last_undegraded":"2024-04-22T08:40:43.896227+0000","last_fullsized":"2024-04-22T08:40:43.896227+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559397+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:20.559397+0000","last_peered":"2024-04-22T08:40:20.559397+0000","last_clean":"2024-04-22T08:40:20.559397+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:20.559397+0000","last_undegraded":"2024-04-22T08:40:20.559397+0000","last_fullsized":"2024-04-22T08:40:20.559397+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44896,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129904640,"data_stored":129897682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117162,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427956,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93943884,"statfs":{"total":96636764160,"available":96198537216,"internally_reserved":0,"allocated":68988928,"data_stored":68674297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.59499999999999997}]}]},{"osd":1,"up_from":33,"seq":141733920924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93856548,"statfs":{"total":96636764160,"available":96109105152,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":527167180},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.443},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.505},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61499999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313391,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.62},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67653632,"data_stored":67653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:57.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:58.231 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:58.232 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:58.252 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-04-22T08:40:57.723829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44896,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1455156,"kb_used_data":124884,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281660364,"statfs":{"total":289910292480,"available":288420212736,"internally_reserved":0,"allocated":127881216,"data_stored":126936827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.233249"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":26423,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-22T08:40:52.800945+0000","last_change":"2024-04-22T08:40:20.014122+0000","last_active":"2024-04-22T08:40:52.800945+0000","last_peered":"2024-04-22T08:40:52.800945+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:52.800945+0000","last_undegraded":"2024-04-22T08:27:56.219112+0000","last_fullsized":"2024-04-22T08:27:56.098408+0000","mapping_epoch":34,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:20.641951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3083,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-711}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20082,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561386+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:20.561386+0000","last_peered":"2024-04-22T08:40:20.561386+0000","last_clean":"2024-04-22T08:40:20.561386+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:20.561386+0000","last_undegraded":"2024-04-22T08:40:20.561386+0000","last_fullsized":"2024-04-22T08:40:20.561386+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19975,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.560795+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:20.560795+0000","last_peered":"2024-04-22T08:40:20.560795+0000","last_clean":"2024-04-22T08:40:20.560795+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:20.560795+0000","last_undegraded":"2024-04-22T08:40:20.560795+0000","last_fullsized":"2024-04-22T08:40:20.560795+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27844,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.644677+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.644677+0000","last_peered":"2024-04-22T08:40:21.644677+0000","last_clean":"2024-04-22T08:40:21.644677+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.644677+0000","last_undegraded":"2024-04-22T08:40:21.644677+0000","last_fullsized":"2024-04-22T08:40:21.644677+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":92,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.561058+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:20.561058+0000","last_peered":"2024-04-22T08:40:20.561058+0000","last_clean":"2024-04-22T08:40:20.561058+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:20.561058+0000","last_undegraded":"2024-04-22T08:40:20.561058+0000","last_fullsized":"2024-04-22T08:40:20.561058+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27664,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:50.801039+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:50.801039+0000","last_peered":"2024-04-22T08:40:50.801039+0000","last_clean":"2024-04-22T08:40:50.801039+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:50.801039+0000","last_undegraded":"2024-04-22T08:40:50.801039+0000","last_fullsized":"2024-04-22T08:40:50.801039+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44896,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":125603840,"data_stored":125596882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117163,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423756,"kb_used_data":63172,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93948084,"statfs":{"total":96636764160,"available":96202838016,"internally_reserved":0,"allocated":64688128,"data_stored":64373497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.437},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313391,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.62},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62246912,"data_stored":62243433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63352832,"data_stored":63352832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:58.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:58.539 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:58.539 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:58.556 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-04-22T08:40:58.493973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1455156,"kb_used_data":124884,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281660364,"statfs":{"total":289910292480,"available":288420212736,"internally_reserved":0,"allocated":127881216,"data_stored":126936827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002825"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27848,"reported_epoch":58,"state":"peering","last_fresh":"2024-04-22T08:40:57.751110+0000","last_change":"2024-04-22T08:40:57.752275+0000","last_active":"2024-04-22T08:40:57.751110+0000","last_peered":"2024-04-22T08:40:57.699320+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:57.751110+0000","last_undegraded":"2024-04-22T08:40:57.751110+0000","last_fullsized":"2024-04-22T08:40:57.751110+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20083,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:21.576477+0000","last_peered":"2024-04-22T08:40:21.576477+0000","last_clean":"2024-04-22T08:40:21.576477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:21.576477+0000","last_undegraded":"2024-04-22T08:40:21.576477+0000","last_fullsized":"2024-04-22T08:40:21.576477+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19976,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576052+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:21.576052+0000","last_peered":"2024-04-22T08:40:21.576052+0000","last_clean":"2024-04-22T08:40:21.576052+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:21.576052+0000","last_undegraded":"2024-04-22T08:40:21.576052+0000","last_fullsized":"2024-04-22T08:40:21.576052+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.645024+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.645024+0000","last_peered":"2024-04-22T08:40:21.645024+0000","last_clean":"2024-04-22T08:40:21.645024+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.645024+0000","last_undegraded":"2024-04-22T08:40:21.645024+0000","last_fullsized":"2024-04-22T08:40:21.645024+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576188+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:21.576188+0000","last_peered":"2024-04-22T08:40:21.576188+0000","last_clean":"2024-04-22T08:40:21.576188+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:21.576188+0000","last_undegraded":"2024-04-22T08:40:21.576188+0000","last_fullsized":"2024-04-22T08:40:21.576188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27664,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:50.801039+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:50.801039+0000","last_peered":"2024-04-22T08:40:50.801039+0000","last_clean":"2024-04-22T08:40:50.801039+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:50.801039+0000","last_undegraded":"2024-04-22T08:40:50.801039+0000","last_fullsized":"2024-04-22T08:40:50.801039+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118616064,"data_stored":118609106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117163,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423756,"kb_used_data":63172,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93948084,"statfs":{"total":96636764160,"available":96202838016,"internally_reserved":0,"allocated":64688128,"data_stored":64373497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.437},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313392,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55259136,"data_stored":55255657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63352832,"data_stored":63352832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:40:58.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:40:58.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 27848, 'reported_epoch': 58, 'state': 'peering', 'last_fresh': '2024-04-22T08:40:57.751110+0000', 'last_change': '2024-04-22T08:40:57.752275+0000', 'last_active': '2024-04-22T08:40:57.751110+0000', 'last_peered': '2024-04-22T08:40:57.699320+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:57.751110+0000', 'last_undegraded': '2024-04-22T08:40:57.751110+0000', 'last_fullsized': '2024-04-22T08:40:57.751110+0000', 'mapping_epoch': 58, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3794, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:40:58.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-04-22T08:40:58.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'19934", 'reported_seq': 27664, 'reported_epoch': 57, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-22T08:40:50.801039+0000', 'last_change': '2024-04-22T08:40:40.903517+0000', 'last_active': '2024-04-22T08:40:50.801039+0000', 'last_peered': '2024-04-22T08:40:50.801039+0000', 'last_clean': '2024-04-22T08:40:50.801039+0000', 'last_became_active': '2024-04-22T08:32:19.148667+0000', 'last_became_peered': '2024-04-22T08:32:19.148667+0000', 'last_unstale': '2024-04-22T08:40:50.801039+0000', 'last_undegraded': '2024-04-22T08:40:50.801039+0000', 'last_fullsized': '2024-04-22T08:40:50.801039+0000', 'mapping_epoch': 38, 'log_start': "37'9900", 'ondisk_log_start': "37'9900", 'created': 12, 'last_epoch_clean': 39, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19932, 'num_write_kb': 39859, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3807, 'num_bytes_recovered': 15577088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-22T08:40:58.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:40:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:58.963 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"peering","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13,"data_bytes":8021,"bytes_used":1490079744,"bytes_avail":288420212736,"bytes_total":289910292480,"inactive_pgs_ratio":0.041666667908430099,"recovering_objects_per_sec":121,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1211,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-22T08:40:42.489689+0000","services":{}},"progress_events":{}} 2024-04-22T08:40:58.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-04-22T08:40:58.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:40:59.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:40:59.280 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:40:59.298 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-04-22T08:40:58.727326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1455156,"kb_used_data":124884,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281660364,"statfs":{"total":289910292480,"available":288420212736,"internally_reserved":0,"allocated":127881216,"data_stored":126936827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.235540"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27848,"reported_epoch":58,"state":"peering","last_fresh":"2024-04-22T08:40:57.751110+0000","last_change":"2024-04-22T08:40:57.752275+0000","last_active":"2024-04-22T08:40:57.751110+0000","last_peered":"2024-04-22T08:40:57.699320+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:57.751110+0000","last_undegraded":"2024-04-22T08:40:57.751110+0000","last_fullsized":"2024-04-22T08:40:57.751110+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20371,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575636+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:20.575636+0000","last_peered":"2024-04-22T08:40:20.575636+0000","last_clean":"2024-04-22T08:40:20.575636+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:20.575636+0000","last_undegraded":"2024-04-22T08:40:20.575636+0000","last_fullsized":"2024-04-22T08:40:20.575636+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20169,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.870411+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:21.870411+0000","last_peered":"2024-04-22T08:40:21.870411+0000","last_clean":"2024-04-22T08:40:21.870411+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:21.870411+0000","last_undegraded":"2024-04-22T08:40:21.870411+0000","last_fullsized":"2024-04-22T08:40:21.870411+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20083,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:21.576477+0000","last_peered":"2024-04-22T08:40:21.576477+0000","last_clean":"2024-04-22T08:40:21.576477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:21.576477+0000","last_undegraded":"2024-04-22T08:40:21.576477+0000","last_fullsized":"2024-04-22T08:40:21.576477+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19976,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576052+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:21.576052+0000","last_peered":"2024-04-22T08:40:21.576052+0000","last_clean":"2024-04-22T08:40:21.576052+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:21.576052+0000","last_undegraded":"2024-04-22T08:40:21.576052+0000","last_fullsized":"2024-04-22T08:40:21.576052+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.574997+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:20.574997+0000","last_peered":"2024-04-22T08:40:20.574997+0000","last_clean":"2024-04-22T08:40:20.574997+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:20.574997+0000","last_undegraded":"2024-04-22T08:40:20.574997+0000","last_fullsized":"2024-04-22T08:40:20.574997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.645024+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.645024+0000","last_peered":"2024-04-22T08:40:21.645024+0000","last_clean":"2024-04-22T08:40:21.645024+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.645024+0000","last_undegraded":"2024-04-22T08:40:21.645024+0000","last_fullsized":"2024-04-22T08:40:21.645024+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576188+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:21.576188+0000","last_peered":"2024-04-22T08:40:21.576188+0000","last_clean":"2024-04-22T08:40:21.576188+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:21.576188+0000","last_undegraded":"2024-04-22T08:40:21.576188+0000","last_fullsized":"2024-04-22T08:40:21.576188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27614,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575105+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:20.575105+0000","last_peered":"2024-04-22T08:40:20.575105+0000","last_clean":"2024-04-22T08:40:20.575105+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:20.575105+0000","last_undegraded":"2024-04-22T08:40:20.575105+0000","last_fullsized":"2024-04-22T08:40:20.575105+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27664,"reported_epoch":57,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T08:40:50.801039+0000","last_change":"2024-04-22T08:40:40.903517+0000","last_active":"2024-04-22T08:40:50.801039+0000","last_peered":"2024-04-22T08:40:50.801039+0000","last_clean":"2024-04-22T08:40:50.801039+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:40:50.801039+0000","last_undegraded":"2024-04-22T08:40:50.801039+0000","last_fullsized":"2024-04-22T08:40:50.801039+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19932,"num_write_kb":39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.575511+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:20.575511+0000","last_peered":"2024-04-22T08:40:20.575511+0000","last_clean":"2024-04-22T08:40:20.575511+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:20.575511+0000","last_undegraded":"2024-04-22T08:40:20.575511+0000","last_fullsized":"2024-04-22T08:40:20.575511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8021,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293691,"num_write_kb":534522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45607,"num_bytes_recovered":38605417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118616064,"data_stored":118609106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117163,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423756,"kb_used_data":63172,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93948084,"statfs":{"total":96636764160,"available":96202838016,"internally_reserved":0,"allocated":64688128,"data_stored":64373497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.437},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.64700000000000002}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313392,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55259136,"data_stored":55255657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63352832,"data_stored":63352832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:02.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:02.610 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:02.611 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:02.631 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-04-22T08:41:02.494942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1450736,"kb_used_data":120464,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281664784,"statfs":{"total":289910292480,"available":288424738816,"internally_reserved":0,"allocated":123355136,"data_stored":122399802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1234,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-19932,"num_write_kb":-39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3096,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27848,"reported_epoch":58,"state":"peering","last_fresh":"2024-04-22T08:40:57.751110+0000","last_change":"2024-04-22T08:40:57.752275+0000","last_active":"2024-04-22T08:40:57.751110+0000","last_peered":"2024-04-22T08:40:57.699320+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:57.751110+0000","last_undegraded":"2024-04-22T08:40:57.751110+0000","last_fullsized":"2024-04-22T08:40:57.751110+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20083,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:21.576477+0000","last_peered":"2024-04-22T08:40:21.576477+0000","last_clean":"2024-04-22T08:40:21.576477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:21.576477+0000","last_undegraded":"2024-04-22T08:40:21.576477+0000","last_fullsized":"2024-04-22T08:40:21.576477+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19976,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576052+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:21.576052+0000","last_peered":"2024-04-22T08:40:21.576052+0000","last_clean":"2024-04-22T08:40:21.576052+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:21.576052+0000","last_undegraded":"2024-04-22T08:40:21.576052+0000","last_fullsized":"2024-04-22T08:40:21.576052+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.645024+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.645024+0000","last_peered":"2024-04-22T08:40:21.645024+0000","last_clean":"2024-04-22T08:40:21.645024+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.645024+0000","last_undegraded":"2024-04-22T08:40:21.645024+0000","last_fullsized":"2024-04-22T08:40:21.645024+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576188+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:21.576188+0000","last_peered":"2024-04-22T08:40:21.576188+0000","last_clean":"2024-04-22T08:40:21.576188+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:21.576188+0000","last_undegraded":"2024-04-22T08:40:21.576188+0000","last_fullsized":"2024-04-22T08:40:21.576188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113950720,"data_stored":113943762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313392,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55259136,"data_stored":55255657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:02.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:02.937 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:02.938 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:02.955 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-04-22T08:41:02.494942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1450736,"kb_used_data":120464,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281664784,"statfs":{"total":289910292480,"available":288424738816,"internally_reserved":0,"allocated":123355136,"data_stored":122399802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1234,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-19932,"num_write_kb":-39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3096,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27848,"reported_epoch":58,"state":"peering","last_fresh":"2024-04-22T08:40:57.751110+0000","last_change":"2024-04-22T08:40:57.752275+0000","last_active":"2024-04-22T08:40:57.751110+0000","last_peered":"2024-04-22T08:40:57.699320+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:57.751110+0000","last_undegraded":"2024-04-22T08:40:57.751110+0000","last_fullsized":"2024-04-22T08:40:57.751110+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20083,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:21.576477+0000","last_peered":"2024-04-22T08:40:21.576477+0000","last_clean":"2024-04-22T08:40:21.576477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:21.576477+0000","last_undegraded":"2024-04-22T08:40:21.576477+0000","last_fullsized":"2024-04-22T08:40:21.576477+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19976,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576052+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:21.576052+0000","last_peered":"2024-04-22T08:40:21.576052+0000","last_clean":"2024-04-22T08:40:21.576052+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:21.576052+0000","last_undegraded":"2024-04-22T08:40:21.576052+0000","last_fullsized":"2024-04-22T08:40:21.576052+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.645024+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.645024+0000","last_peered":"2024-04-22T08:40:21.645024+0000","last_clean":"2024-04-22T08:40:21.645024+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.645024+0000","last_undegraded":"2024-04-22T08:40:21.645024+0000","last_fullsized":"2024-04-22T08:40:21.645024+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576188+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:21.576188+0000","last_peered":"2024-04-22T08:40:21.576188+0000","last_clean":"2024-04-22T08:40:21.576188+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:21.576188+0000","last_undegraded":"2024-04-22T08:40:21.576188+0000","last_fullsized":"2024-04-22T08:40:21.576188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113950720,"data_stored":113943762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313392,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55259136,"data_stored":55255657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:02.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-04-22T08:41:02.957 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'20168", 'reported_seq': 27848, 'reported_epoch': 58, 'state': 'peering', 'last_fresh': '2024-04-22T08:40:57.751110+0000', 'last_change': '2024-04-22T08:40:57.752275+0000', 'last_active': '2024-04-22T08:40:57.751110+0000', 'last_peered': '2024-04-22T08:40:57.699320+0000', 'last_clean': '2024-04-22T08:27:54.065448+0000', 'last_became_active': '2024-04-22T08:27:56.746365+0000', 'last_became_peered': '2024-04-22T08:27:56.746365+0000', 'last_unstale': '2024-04-22T08:40:57.751110+0000', 'last_undegraded': '2024-04-22T08:40:57.751110+0000', 'last_fullsized': '2024-04-22T08:40:57.751110+0000', 'mapping_epoch': 58, 'log_start': "37'10100", 'ondisk_log_start': "37'10100", 'created': 12, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'last_clean_scrub_stamp': '2024-04-22T08:26:27.267649+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20168, 'num_write_kb': 40336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3794, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-22T08:41:02.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T08:41:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:03.309 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":898,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1713774474,"num_in_osds":3,"osd_in_since":1713774465,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"peering","count":1}],"num_pgs":24,"num_pools":2,"num_objects":11,"data_bytes":6787,"bytes_used":1485553664,"bytes_avail":288424738816,"bytes_total":289910292480,"inactive_pgs_ratio":0.041666667908430099},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-22T08:40:42.489689+0000","services":{}},"progress_events":{}} 2024-04-22T08:41:03.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:no progress seen, keeping timeout for now 2024-04-22T08:41:03.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:03.622 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:03.623 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:03.639 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-04-22T08:41:02.494942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1450736,"kb_used_data":120464,"kb_used_omap":12,"kb_used_meta":1330227,"kb_avail":281664784,"statfs":{"total":289910292480,"available":288424738816,"internally_reserved":0,"allocated":123355136,"data_stored":122399802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1362152558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1234,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-19932,"num_write_kb":-39859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3096,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27848,"reported_epoch":58,"state":"peering","last_fresh":"2024-04-22T08:40:57.751110+0000","last_change":"2024-04-22T08:40:57.752275+0000","last_active":"2024-04-22T08:40:57.751110+0000","last_peered":"2024-04-22T08:40:57.699320+0000","last_clean":"2024-04-22T08:27:54.065448+0000","last_became_active":"2024-04-22T08:27:56.746365+0000","last_became_peered":"2024-04-22T08:27:56.746365+0000","last_unstale":"2024-04-22T08:40:57.751110+0000","last_undegraded":"2024-04-22T08:40:57.751110+0000","last_fullsized":"2024-04-22T08:40:57.751110+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:01:00.903003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20424,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576298+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:21.576298+0000","last_peered":"2024-04-22T08:40:21.576298+0000","last_clean":"2024-04-22T08:40:21.576298+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:21.576298+0000","last_undegraded":"2024-04-22T08:40:21.576298+0000","last_fullsized":"2024-04-22T08:40:21.576298+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20083,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576477+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:21.576477+0000","last_peered":"2024-04-22T08:40:21.576477+0000","last_clean":"2024-04-22T08:40:21.576477+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:21.576477+0000","last_undegraded":"2024-04-22T08:40:21.576477+0000","last_fullsized":"2024-04-22T08:40:21.576477+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28007,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558672+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:20.558672+0000","last_peered":"2024-04-22T08:40:20.558672+0000","last_clean":"2024-04-22T08:40:20.558672+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:20.558672+0000","last_undegraded":"2024-04-22T08:40:20.558672+0000","last_fullsized":"2024-04-22T08:40:20.558672+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27624,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558395+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:20.558395+0000","last_peered":"2024-04-22T08:40:20.558395+0000","last_clean":"2024-04-22T08:40:20.558395+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:20.558395+0000","last_undegraded":"2024-04-22T08:40:20.558395+0000","last_fullsized":"2024-04-22T08:40:20.558395+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19976,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576052+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:21.576052+0000","last_peered":"2024-04-22T08:40:21.576052+0000","last_clean":"2024-04-22T08:40:21.576052+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:21.576052+0000","last_undegraded":"2024-04-22T08:40:21.576052+0000","last_fullsized":"2024-04-22T08:40:21.576052+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558127+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:20.558127+0000","last_peered":"2024-04-22T08:40:20.558127+0000","last_clean":"2024-04-22T08:40:20.558127+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:20.558127+0000","last_undegraded":"2024-04-22T08:40:20.558127+0000","last_fullsized":"2024-04-22T08:40:20.558127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20699,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558627+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:20.558627+0000","last_peered":"2024-04-22T08:40:20.558627+0000","last_clean":"2024-04-22T08:40:20.558627+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:20.558627+0000","last_undegraded":"2024-04-22T08:40:20.558627+0000","last_fullsized":"2024-04-22T08:40:20.558627+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27845,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.645024+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:21.645024+0000","last_peered":"2024-04-22T08:40:21.645024+0000","last_clean":"2024-04-22T08:40:21.645024+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:21.645024+0000","last_undegraded":"2024-04-22T08:40:21.645024+0000","last_fullsized":"2024-04-22T08:40:21.645024+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":93,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576188+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:21.576188+0000","last_peered":"2024-04-22T08:40:21.576188+0000","last_clean":"2024-04-22T08:40:21.576188+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:21.576188+0000","last_undegraded":"2024-04-22T08:40:21.576188+0000","last_fullsized":"2024-04-22T08:40:21.576188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.576393+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:21.576393+0000","last_peered":"2024-04-22T08:40:21.576393+0000","last_clean":"2024-04-22T08:40:21.576393+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:21.576393+0000","last_undegraded":"2024-04-22T08:40:21.576393+0000","last_fullsized":"2024-04-22T08:40:21.576393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20912,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559637+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:20.559637+0000","last_peered":"2024-04-22T08:40:20.559637+0000","last_clean":"2024-04-22T08:40:20.559637+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:20.559637+0000","last_undegraded":"2024-04-22T08:40:20.559637+0000","last_fullsized":"2024-04-22T08:40:20.559637+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577030+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:21.577030+0000","last_peered":"2024-04-22T08:40:21.577030+0000","last_clean":"2024-04-22T08:40:21.577030+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:21.577030+0000","last_undegraded":"2024-04-22T08:40:21.577030+0000","last_fullsized":"2024-04-22T08:40:21.577030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20440,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558960+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:20.558960+0000","last_peered":"2024-04-22T08:40:20.558960+0000","last_clean":"2024-04-22T08:40:20.558960+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:20.558960+0000","last_undegraded":"2024-04-22T08:40:20.558960+0000","last_fullsized":"2024-04-22T08:40:20.558960+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20666,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.559052+0000","last_change":"2024-04-22T08:37:55.208193+0000","last_active":"2024-04-22T08:40:20.559052+0000","last_peered":"2024-04-22T08:40:20.559052+0000","last_clean":"2024-04-22T08:40:20.559052+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:20.559052+0000","last_undegraded":"2024-04-22T08:40:20.559052+0000","last_fullsized":"2024-04-22T08:40:20.559052+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20036,"num_write_kb":40070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577300+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:21.577300+0000","last_peered":"2024-04-22T08:40:21.577300+0000","last_clean":"2024-04-22T08:40:21.577300+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:21.577300+0000","last_undegraded":"2024-04-22T08:40:21.577300+0000","last_fullsized":"2024-04-22T08:40:21.577300+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T08:40:21.577237+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:21.577237+0000","last_peered":"2024-04-22T08:40:21.577237+0000","last_clean":"2024-04-22T08:40:21.577237+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:21.577237+0000","last_undegraded":"2024-04-22T08:40:21.577237+0000","last_fullsized":"2024-04-22T08:40:21.577237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20739,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-22T08:40:20.558597+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:20.558597+0000","last_peered":"2024-04-22T08:40:20.558597+0000","last_clean":"2024-04-22T08:40:20.558597+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:20.558597+0000","last_undegraded":"2024-04-22T08:40:20.558597+0000","last_fullsized":"2024-04-22T08:40:20.558597+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":273759,"num_write_kb":494663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41800,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113950720,"data_stored":113943762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519388,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93852452,"statfs":{"total":96636764160,"available":96104910848,"internally_reserved":0,"allocated":479232,"data_stored":161122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":531361484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.69199999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313392,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512012,"kb_used_data":61244,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93859828,"statfs":{"total":96636764160,"available":96112463872,"internally_reserved":0,"allocated":62713856,"data_stored":62402208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":461565651},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55259136,"data_stored":55255657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:06.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:06.951 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:06.972 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-22T08:41:06.495954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398168,"kb_used_data":59712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281717352,"statfs":{"total":289910292480,"available":288478568448,"internally_reserved":0,"allocated":61145088,"data_stored":60167864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2468,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-39968,"num_write_kb":-79929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6805,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.772125"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757011+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:57.757011+0000","last_peered":"2024-04-22T08:40:57.757011+0000","last_clean":"2024-04-22T08:40:57.757011+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:57.757011+0000","last_undegraded":"2024-04-22T08:40:57.757011+0000","last_fullsized":"2024-04-22T08:40:57.757011+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757630+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:57.757630+0000","last_peered":"2024-04-22T08:40:57.757630+0000","last_clean":"2024-04-22T08:40:57.757630+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:57.757630+0000","last_undegraded":"2024-04-22T08:40:57.757630+0000","last_fullsized":"2024-04-22T08:40:57.757630+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27846,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737178+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:57.737178+0000","last_peered":"2024-04-22T08:40:57.737178+0000","last_clean":"2024-04-22T08:40:57.737178+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:57.737178+0000","last_undegraded":"2024-04-22T08:40:57.737178+0000","last_fullsized":"2024-04-22T08:40:57.737178+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20742,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731945+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:58.731945+0000","last_peered":"2024-04-22T08:40:58.731945+0000","last_clean":"2024-04-22T08:40:58.731945+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:58.731945+0000","last_undegraded":"2024-04-22T08:40:58.731945+0000","last_fullsized":"2024-04-22T08:40:58.731945+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58695680,"data_stored":58688722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:06.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-04-22T08:41:06.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-04-22T08:41:06.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:07.258 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:07.258 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:07.276 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-22T08:41:06.495954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398168,"kb_used_data":59712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281717352,"statfs":{"total":289910292480,"available":288478568448,"internally_reserved":0,"allocated":61145088,"data_stored":60167864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2468,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-39968,"num_write_kb":-79929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6805,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.772125"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757011+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:57.757011+0000","last_peered":"2024-04-22T08:40:57.757011+0000","last_clean":"2024-04-22T08:40:57.757011+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:57.757011+0000","last_undegraded":"2024-04-22T08:40:57.757011+0000","last_fullsized":"2024-04-22T08:40:57.757011+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757630+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:57.757630+0000","last_peered":"2024-04-22T08:40:57.757630+0000","last_clean":"2024-04-22T08:40:57.757630+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:57.757630+0000","last_undegraded":"2024-04-22T08:40:57.757630+0000","last_fullsized":"2024-04-22T08:40:57.757630+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27846,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737178+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:57.737178+0000","last_peered":"2024-04-22T08:40:57.737178+0000","last_clean":"2024-04-22T08:40:57.737178+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:57.737178+0000","last_undegraded":"2024-04-22T08:40:57.737178+0000","last_fullsized":"2024-04-22T08:40:57.737178+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20742,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731945+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:58.731945+0000","last_peered":"2024-04-22T08:40:58.731945+0000","last_clean":"2024-04-22T08:40:58.731945+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:58.731945+0000","last_undegraded":"2024-04-22T08:40:58.731945+0000","last_fullsized":"2024-04-22T08:40:58.731945+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58695680,"data_stored":58688722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:07.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:07.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:07.576 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:07.595 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-22T08:41:06.495954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398168,"kb_used_data":59712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281717352,"statfs":{"total":289910292480,"available":288478568448,"internally_reserved":0,"allocated":61145088,"data_stored":60167864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2468,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-39968,"num_write_kb":-79929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6805,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.772125"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757011+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:57.757011+0000","last_peered":"2024-04-22T08:40:57.757011+0000","last_clean":"2024-04-22T08:40:57.757011+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:57.757011+0000","last_undegraded":"2024-04-22T08:40:57.757011+0000","last_fullsized":"2024-04-22T08:40:57.757011+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757630+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:57.757630+0000","last_peered":"2024-04-22T08:40:57.757630+0000","last_clean":"2024-04-22T08:40:57.757630+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:57.757630+0000","last_undegraded":"2024-04-22T08:40:57.757630+0000","last_fullsized":"2024-04-22T08:40:57.757630+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27846,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737178+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:57.737178+0000","last_peered":"2024-04-22T08:40:57.737178+0000","last_clean":"2024-04-22T08:40:57.737178+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:57.737178+0000","last_undegraded":"2024-04-22T08:40:57.737178+0000","last_fullsized":"2024-04-22T08:40:57.737178+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20742,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731945+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:58.731945+0000","last_peered":"2024-04-22T08:40:58.731945+0000","last_clean":"2024-04-22T08:40:58.731945+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:58.731945+0000","last_undegraded":"2024-04-22T08:40:58.731945+0000","last_fullsized":"2024-04-22T08:40:58.731945+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58695680,"data_stored":58688722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:07.597 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-04-22T08:41:07.597 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T08:41:07.607 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T08:41:07.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:07.895 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:07.914 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-22T08:41:06.495954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398168,"kb_used_data":59712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281717352,"statfs":{"total":289910292480,"available":288478568448,"internally_reserved":0,"allocated":61145088,"data_stored":60167864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2468,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-39968,"num_write_kb":-79929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6805,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.772125"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757011+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:57.757011+0000","last_peered":"2024-04-22T08:40:57.757011+0000","last_clean":"2024-04-22T08:40:57.757011+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:57.757011+0000","last_undegraded":"2024-04-22T08:40:57.757011+0000","last_fullsized":"2024-04-22T08:40:57.757011+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757630+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:57.757630+0000","last_peered":"2024-04-22T08:40:57.757630+0000","last_clean":"2024-04-22T08:40:57.757630+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:57.757630+0000","last_undegraded":"2024-04-22T08:40:57.757630+0000","last_fullsized":"2024-04-22T08:40:57.757630+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27846,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737178+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:57.737178+0000","last_peered":"2024-04-22T08:40:57.737178+0000","last_clean":"2024-04-22T08:40:57.737178+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:57.737178+0000","last_undegraded":"2024-04-22T08:40:57.737178+0000","last_fullsized":"2024-04-22T08:40:57.737178+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20742,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731945+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:58.731945+0000","last_peered":"2024-04-22T08:40:58.731945+0000","last_clean":"2024-04-22T08:40:58.731945+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:58.731945+0000","last_undegraded":"2024-04-22T08:40:58.731945+0000","last_fullsized":"2024-04-22T08:40:58.731945+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58695680,"data_stored":58688722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:07.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:08.214 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:08.215 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:08.232 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-22T08:41:06.495954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1398168,"kb_used_data":59712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281717352,"statfs":{"total":289910292480,"available":288478568448,"internally_reserved":0,"allocated":61145088,"data_stored":60167864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2468,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-39968,"num_write_kb":-79929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6805,"num_bytes_recovered":-15577088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.772125"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757011+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:57.757011+0000","last_peered":"2024-04-22T08:40:57.757011+0000","last_clean":"2024-04-22T08:40:57.757011+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:57.757011+0000","last_undegraded":"2024-04-22T08:40:57.757011+0000","last_fullsized":"2024-04-22T08:40:57.757011+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757630+0000","last_change":"2024-04-22T08:39:32.839760+0000","last_active":"2024-04-22T08:40:57.757630+0000","last_peered":"2024-04-22T08:40:57.757630+0000","last_clean":"2024-04-22T08:40:57.757630+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:40:57.757630+0000","last_undegraded":"2024-04-22T08:40:57.757630+0000","last_fullsized":"2024-04-22T08:40:57.757630+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":798,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20161,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27846,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737178+0000","last_change":"2024-04-22T08:40:21.644677+0000","last_active":"2024-04-22T08:40:57.737178+0000","last_peered":"2024-04-22T08:40:57.737178+0000","last_clean":"2024-04-22T08:40:57.737178+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:40:57.737178+0000","last_undegraded":"2024-04-22T08:40:57.737178+0000","last_fullsized":"2024-04-22T08:40:57.737178+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":40360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27678,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897209+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897209+0000","last_peered":"2024-04-22T08:41:00.897209+0000","last_clean":"2024-04-22T08:41:00.897209+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897209+0000","last_undegraded":"2024-04-22T08:41:00.897209+0000","last_fullsized":"2024-04-22T08:41:00.897209+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20742,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731945+0000","last_change":"2024-04-22T08:38:57.298839+0000","last_active":"2024-04-22T08:40:58.731945+0000","last_peered":"2024-04-22T08:40:58.731945+0000","last_clean":"2024-04-22T08:40:58.731945+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:40:58.731945+0000","last_undegraded":"2024-04-22T08:40:58.731945+0000","last_fullsized":"2024-04-22T08:40:58.731945+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":2649,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3889,"num_bytes_recovered":4713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253723,"num_write_kb":454593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38091,"num_bytes_recovered":23028329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58695680,"data_stored":58688722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419336,"kb_used_data":58752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93952504,"statfs":{"total":96636764160,"available":96207364096,"internally_reserved":0,"allocated":60162048,"data_stored":59836472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.61599999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.61799999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58687488,"data_stored":58687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:08.234 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T08:41:08.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:08.520 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:08.521 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:08.537 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-04-22T08:41:08.496387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393368,"kb_used_data":54912,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281722152,"statfs":{"total":289910292480,"available":288483483648,"internally_reserved":0,"allocated":56229888,"data_stored":55252664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6787,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-100312,"num_write_kb":-200592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19076,"num_bytes_recovered":-15585897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002414"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27627,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731657+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:58.731657+0000","last_peered":"2024-04-22T08:40:58.731657+0000","last_clean":"2024-04-22T08:40:58.731657+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:58.731657+0000","last_undegraded":"2024-04-22T08:40:58.731657+0000","last_fullsized":"2024-04-22T08:40:58.731657+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:05.176447+0000","last_change":"2024-04-22T08:41:05.176670+0000","last_active":"2024-04-22T08:41:05.176447+0000","last_peered":"2024-04-22T08:41:05.176447+0000","last_clean":"2024-04-22T08:41:05.176447+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:41:05.176447+0000","last_undegraded":"2024-04-22T08:41:05.176447+0000","last_fullsized":"2024-04-22T08:41:05.176447+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20221","last_scrub_stamp":"2024-04-22T08:41:05.176408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:05.176408+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.237730+0000","scrub_duration":0.041926848000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27858,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:04.252232+0000","last_change":"2024-04-22T08:41:04.252415+0000","last_active":"2024-04-22T08:41:04.252232+0000","last_peered":"2024-04-22T08:41:04.252232+0000","last_clean":"2024-04-22T08:41:04.252232+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:41:04.252232+0000","last_undegraded":"2024-04-22T08:41:04.252232+0000","last_fullsized":"2024-04-22T08:41:04.252232+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20186","last_scrub_stamp":"2024-04-22T08:41:04.252203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:04.252203+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:32.140027+0000","scrub_duration":0.065609869000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27679,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897430+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897430+0000","last_peered":"2024-04-22T08:41:00.897430+0000","last_clean":"2024-04-22T08:41:00.897430+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897430+0000","last_undegraded":"2024-04-22T08:41:00.897430+0000","last_fullsized":"2024-04-22T08:41:00.897430+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20754,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:04.194436+0000","last_change":"2024-04-22T08:41:04.194624+0000","last_active":"2024-04-22T08:41:04.194436+0000","last_peered":"2024-04-22T08:41:04.194436+0000","last_clean":"2024-04-22T08:41:04.194436+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:41:04.194436+0000","last_undegraded":"2024-04-22T08:41:04.194436+0000","last_fullsized":"2024-04-22T08:41:04.194436+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20002","last_scrub_stamp":"2024-04-22T08:41:04.194401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:04.194401+0000","objects_scrubbed":1,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:13.750423+0000","scrub_duration":0.077576006000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53780480,"data_stored":53773522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117165,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414536,"kb_used_data":53952,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93957304,"statfs":{"total":96636764160,"available":96212279296,"internally_reserved":0,"allocated":55246848,"data_stored":54921272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.622}]}]},{"osd":1,"up_from":33,"seq":141733920927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.495},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.64100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313394,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.501},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53772288,"data_stored":53772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:08.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:41:08.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:08.811 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":59,"fsid":"7ee9519b-73a7-4b76-a33e-63cd80692d01","created":"2024-04-22T08:26:05.080988+0000","modified":"2024-04-22T08:40:58.722456+0000","last_up_change":"2024-04-22T08:27:54.066710+0000","last_in_change":"2024-04-22T08:27:45.797437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T08:26:13.951550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-04-22T08:26:26.331332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"318c137f-1dc4-4513-bc1a-acaadc0ab20c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":21,"up_thru":58,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6800","nonce":29827}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6801","nonce":29827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6803","nonce":29827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6802","nonce":29827}]},"public_addr":"172.21.15.49:6800/29827","cluster_addr":"172.21.15.49:6801/29827","heartbeat_back_addr":"172.21.15.49:6803/29827","heartbeat_front_addr":"172.21.15.49:6802/29827","state":["exists","up"]},{"osd":1,"uuid":"f2692c19-8d2d-4d9d-8de5-d0d885c8f379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":22,"up_from":33,"up_thru":54,"down_at":23,"lost_at":23,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6808","nonce":34075}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6809","nonce":34075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6811","nonce":34075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6810","nonce":34075}]},"public_addr":"172.21.15.49:6808/34075","cluster_addr":"172.21.15.49:6809/34075","heartbeat_back_addr":"172.21.15.49:6811/34075","heartbeat_front_addr":"172.21.15.49:6810/34075","state":["exists","up"]},{"osd":2,"uuid":"2135f37a-1d7c-4e8e-a42f-09f7bfcb8ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":14,"up_from":27,"up_thru":49,"down_at":15,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6804","nonce":31087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6805","nonce":31087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6807","nonce":31087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.49:6806","nonce":31087}]},"public_addr":"172.21.15.49:6804/31087","cluster_addr":"172.21.15.49:6805/31087","heartbeat_back_addr":"172.21.15.49:6807/31087","heartbeat_front_addr":"172.21.15.49:6806/31087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.496890+0000","dead_epoch":18},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.885475+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:26:11.582988+0000","dead_epoch":15}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:41:09.829 INFO:tasks.ceph:Scrubbing osd.0 2024-04-22T08:41:09.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:41:09.987 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:41:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:41:09.988 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:41:10.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-22T08:41:10.301 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 0 to deep-scrub 2024-04-22T08:41:10.320 INFO:tasks.ceph:Scrubbing osd.1 2024-04-22T08:41:10.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:41:10.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-22T08:41:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 1 to deep-scrub 2024-04-22T08:41:10.788 INFO:tasks.ceph:Scrubbing osd.2 2024-04-22T08:41:10.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:41:10.931 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-22T08:41:10.932 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:41:10.932 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-22T08:41:10.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-22T08:41:11.233 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 2 to deep-scrub 2024-04-22T08:41:11.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:11.534 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:11.553 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-04-22T08:41:10.496987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393368,"kb_used_data":54912,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281722152,"statfs":{"total":289910292480,"available":288483483648,"internally_reserved":0,"allocated":56229888,"data_stored":55252664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6787,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-100312,"num_write_kb":-200592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19076,"num_bytes_recovered":-15585897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.769661"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27856,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.841521+0000","last_change":"2024-04-22T08:40:58.841521+0000","last_active":"2024-04-22T08:40:58.841521+0000","last_peered":"2024-04-22T08:40:58.841521+0000","last_clean":"2024-04-22T08:40:58.841521+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:40:58.841521+0000","last_undegraded":"2024-04-22T08:40:58.841521+0000","last_fullsized":"2024-04-22T08:40:58.841521+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:51.180317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20373,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727838+0000","last_change":"2024-04-22T08:27:40.690815+0000","last_active":"2024-04-22T08:40:57.727838+0000","last_peered":"2024-04-22T08:40:57.727838+0000","last_clean":"2024-04-22T08:40:57.727838+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:40:57.727838+0000","last_undegraded":"2024-04-22T08:40:57.727838+0000","last_fullsized":"2024-04-22T08:40:57.727838+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3009","last_scrub_stamp":"2024-04-22T08:27:40.690572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.690572+0000","objects_scrubbed":2938,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:59:09.872377+0000","scrub_duration":1.8119097399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20426,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757463+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:57.757463+0000","last_peered":"2024-04-22T08:40:57.757463+0000","last_clean":"2024-04-22T08:40:57.757463+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:57.757463+0000","last_undegraded":"2024-04-22T08:40:57.757463+0000","last_fullsized":"2024-04-22T08:40:57.757463+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:31.788473+0000","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20171,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727368+0000","last_change":"2024-04-22T08:27:06.399040+0000","last_active":"2024-04-22T08:40:57.727368+0000","last_peered":"2024-04-22T08:40:57.727368+0000","last_clean":"2024-04-22T08:40:57.727368+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:40:57.727368+0000","last_undegraded":"2024-04-22T08:40:57.727368+0000","last_fullsized":"2024-04-22T08:40:57.727368+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:42.410538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20084,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.737500+0000","last_change":"2024-04-22T08:27:40.134611+0000","last_active":"2024-04-22T08:40:57.737500+0000","last_peered":"2024-04-22T08:40:57.737500+0000","last_clean":"2024-04-22T08:40:57.737500+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:40:57.737500+0000","last_undegraded":"2024-04-22T08:40:57.737500+0000","last_fullsized":"2024-04-22T08:40:57.737500+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2946","last_scrub_stamp":"2024-04-22T08:27:40.133871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:40.133871+0000","objects_scrubbed":2901,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:34:07.131254+0000","scrub_duration":1.577913466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28009,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757317+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:57.757317+0000","last_peered":"2024-04-22T08:40:57.757317+0000","last_clean":"2024-04-22T08:40:57.757317+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:57.757317+0000","last_undegraded":"2024-04-22T08:40:57.757317+0000","last_fullsized":"2024-04-22T08:40:57.757317+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:34.215070+0000","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27627,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731657+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:40:58.731657+0000","last_peered":"2024-04-22T08:40:58.731657+0000","last_clean":"2024-04-22T08:40:58.731657+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:40:58.731657+0000","last_undegraded":"2024-04-22T08:40:58.731657+0000","last_fullsized":"2024-04-22T08:40:58.731657+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:30:00.039079+0000","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19977,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.736894+0000","last_change":"2024-04-22T08:27:47.201557+0000","last_active":"2024-04-22T08:40:57.736894+0000","last_peered":"2024-04-22T08:40:57.736894+0000","last_clean":"2024-04-22T08:40:57.736894+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:40:57.736894+0000","last_undegraded":"2024-04-22T08:40:57.736894+0000","last_fullsized":"2024-04-22T08:40:57.736894+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3352","last_scrub_stamp":"2024-04-22T08:27:47.201346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:47.201346+0000","objects_scrubbed":3350,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:19:29.035715+0000","scrub_duration":0.59916720700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.756653+0000","last_change":"2024-04-22T08:27:56.106880+0000","last_active":"2024-04-22T08:40:57.756653+0000","last_peered":"2024-04-22T08:40:57.756653+0000","last_clean":"2024-04-22T08:40:57.756653+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:40:57.756653+0000","last_undegraded":"2024-04-22T08:40:57.756653+0000","last_fullsized":"2024-04-22T08:40:57.756653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:32.386348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:05.176447+0000","last_change":"2024-04-22T08:41:05.176670+0000","last_active":"2024-04-22T08:41:05.176447+0000","last_peered":"2024-04-22T08:41:05.176447+0000","last_clean":"2024-04-22T08:41:05.176447+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:41:05.176447+0000","last_undegraded":"2024-04-22T08:41:05.176447+0000","last_fullsized":"2024-04-22T08:41:05.176447+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20221","last_scrub_stamp":"2024-04-22T08:41:05.176408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:05.176408+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.237730+0000","scrub_duration":0.041926848000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726745+0000","last_change":"2024-04-22T08:27:06.396348+0000","last_active":"2024-04-22T08:40:57.726745+0000","last_peered":"2024-04-22T08:40:57.726745+0000","last_clean":"2024-04-22T08:40:57.726745+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:40:57.726745+0000","last_undegraded":"2024-04-22T08:40:57.726745+0000","last_fullsized":"2024-04-22T08:40:57.726745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:13:03.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27858,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:04.252232+0000","last_change":"2024-04-22T08:41:04.252415+0000","last_active":"2024-04-22T08:41:04.252232+0000","last_peered":"2024-04-22T08:41:04.252232+0000","last_clean":"2024-04-22T08:41:04.252232+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:41:04.252232+0000","last_undegraded":"2024-04-22T08:41:04.252232+0000","last_fullsized":"2024-04-22T08:41:04.252232+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20186","last_scrub_stamp":"2024-04-22T08:41:04.252203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:04.252203+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:32.140027+0000","scrub_duration":0.065609869000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":94,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.733109+0000","last_change":"2024-04-22T08:27:56.249956+0000","last_active":"2024-04-22T08:40:57.733109+0000","last_peered":"2024-04-22T08:40:57.733109+0000","last_clean":"2024-04-22T08:40:57.733109+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:40:57.733109+0000","last_undegraded":"2024-04-22T08:40:57.733109+0000","last_fullsized":"2024-04-22T08:40:57.733109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:02:11.599765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27616,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.726791+0000","last_change":"2024-04-22T08:37:18.676000+0000","last_active":"2024-04-22T08:40:57.726791+0000","last_peered":"2024-04-22T08:40:57.726791+0000","last_clean":"2024-04-22T08:40:57.726791+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:40:57.726791+0000","last_undegraded":"2024-04-22T08:40:57.726791+0000","last_fullsized":"2024-04-22T08:40:57.726791+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3623","last_scrub_stamp":"2024-04-22T08:27:52.002961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.002961+0000","objects_scrubbed":3541,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:03.374253+0000","scrub_duration":2.1184628600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757394+0000","last_change":"2024-04-22T08:27:56.216998+0000","last_active":"2024-04-22T08:40:57.757394+0000","last_peered":"2024-04-22T08:40:57.757394+0000","last_clean":"2024-04-22T08:40:57.757394+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:40:57.757394+0000","last_undegraded":"2024-04-22T08:40:57.757394+0000","last_fullsized":"2024-04-22T08:40:57.757394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:21.320951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27679,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:00.897430+0000","last_change":"2024-04-22T08:41:00.897430+0000","last_active":"2024-04-22T08:41:00.897430+0000","last_peered":"2024-04-22T08:41:00.897430+0000","last_clean":"2024-04-22T08:41:00.897430+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:00.897430+0000","last_undegraded":"2024-04-22T08:41:00.897430+0000","last_fullsized":"2024-04-22T08:41:00.897430+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:00.897171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:00.897171+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:42:49.014478+0000","scrub_duration":0.058619349000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20914,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757625+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:57.757625+0000","last_peered":"2024-04-22T08:40:57.757625+0000","last_clean":"2024-04-22T08:40:57.757625+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:57.757625+0000","last_undegraded":"2024-04-22T08:40:57.757625+0000","last_fullsized":"2024-04-22T08:40:57.757625+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757481+0000","last_change":"2024-04-22T08:27:56.353404+0000","last_active":"2024-04-22T08:40:57.757481+0000","last_peered":"2024-04-22T08:40:57.757481+0000","last_clean":"2024-04-22T08:40:57.757481+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:40:57.757481+0000","last_undegraded":"2024-04-22T08:40:57.757481+0000","last_fullsized":"2024-04-22T08:40:57.757481+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:29:28.925948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20442,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:09.817421+0000","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.727871+0000","last_change":"2024-04-22T08:27:06.398585+0000","last_active":"2024-04-22T08:40:57.727871+0000","last_peered":"2024-04-22T08:40:57.727871+0000","last_clean":"2024-04-22T08:40:57.727871+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:40:57.727871+0000","last_undegraded":"2024-04-22T08:40:57.727871+0000","last_fullsized":"2024-04-22T08:40:57.727871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:29:01.175921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20680,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300738+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300738+0000","last_peered":"2024-04-22T08:40:59.300738+0000","last_clean":"2024-04-22T08:40:59.300738+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300738+0000","last_undegraded":"2024-04-22T08:40:59.300738+0000","last_fullsized":"2024-04-22T08:40:59.300738+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:05:32.232521+0000","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757711+0000","last_change":"2024-04-22T08:27:56.367049+0000","last_active":"2024-04-22T08:40:57.757711+0000","last_peered":"2024-04-22T08:40:57.757711+0000","last_clean":"2024-04-22T08:40:57.757711+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:40:57.757711+0000","last_undegraded":"2024-04-22T08:40:57.757711+0000","last_fullsized":"2024-04-22T08:40:57.757711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:16:50.404647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T08:40:57.757494+0000","last_change":"2024-04-22T08:27:56.102298+0000","last_active":"2024-04-22T08:40:57.757494+0000","last_peered":"2024-04-22T08:40:57.757494+0000","last_clean":"2024-04-22T08:40:57.757494+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:40:57.757494+0000","last_undegraded":"2024-04-22T08:40:57.757494+0000","last_fullsized":"2024-04-22T08:40:57.757494+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:14.194929+0000","last_clean_scrub_stamp":"2024-04-22T08:26:14.194929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:21.962946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20754,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:04.194436+0000","last_change":"2024-04-22T08:41:04.194624+0000","last_active":"2024-04-22T08:41:04.194436+0000","last_peered":"2024-04-22T08:41:04.194436+0000","last_clean":"2024-04-22T08:41:04.194436+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:41:04.194436+0000","last_undegraded":"2024-04-22T08:41:04.194436+0000","last_fullsized":"2024-04-22T08:41:04.194436+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20002","last_scrub_stamp":"2024-04-22T08:41:04.194401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:04.194401+0000","objects_scrubbed":1,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:13.750423+0000","scrub_duration":0.077576006000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53780480,"data_stored":53773522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117165,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414536,"kb_used_data":53952,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93957304,"statfs":{"total":96636764160,"available":96212279296,"internally_reserved":0,"allocated":55246848,"data_stored":54921272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Apr 22 08:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.82799999999999996,"15min":1.0369999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.83499999999999996,"5min":15.6,"15min":15.6},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.97399999999999998,"15min":1.0269999999999999},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":0.93700000000000006,"5min":6.6239999999999997,"15min":18.32},"last":0.622}]}]},{"osd":1,"up_from":33,"seq":141733920927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.56399999999999995,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.495},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.87,"15min":0.876},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":0.67700000000000005,"5min":6.1589999999999998,"15min":6.8860000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.61199999999999999,"15min":1.1519999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52700000000000002,"5min":4.8929999999999998,"15min":15.989000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.67000000000000004,"15min":1.0940000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":4.4489999999999998,"15min":20.149000000000001},"last":0.64100000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313394,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.91200000000000003,"15min":0.98299999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.48899999999999999,"5min":10.972,"15min":11.368},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.79700000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56799999999999995,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.501},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53772288,"data_stored":53772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:11.557 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-22T08:26:27.267649+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=27, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.557 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-22T08:27:40.690572+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=40, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.557 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-22T08:27:44.400987+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=44, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-22T08:26:27.267649+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=27, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-22T08:27:40.133871+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=40, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-22T08:27:42.225197+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=42, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-22T08:27:52.930922+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=52, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-22T08:27:47.201346+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=47, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-22T08:41:05.176408+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=5, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.558 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-22T08:41:04.252203+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=4, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-22T08:27:52.002961+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=52, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-22T08:41:00.897171+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=0, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-22T08:26:27.267649+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=27, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-22T08:27:53.920523+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=53, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-22T08:40:59.300717+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=40, tm_sec=59, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.559 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.560 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-22T08:26:14.194929+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=14, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.560 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-22T08:41:04.194401+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=4, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:11.560 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T08:41:31.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:31.870 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:31.871 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:31.888 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-04-22T08:41:30.502156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374168,"kb_used_data":35712,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281741352,"statfs":{"total":289910292480,"available":288503144448,"internally_reserved":0,"allocated":36569088,"data_stored":35591864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27868,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.328340+0000","last_change":"2024-04-22T08:41:26.328540+0000","last_active":"2024-04-22T08:41:26.328340+0000","last_peered":"2024-04-22T08:41:26.328340+0000","last_clean":"2024-04-22T08:41:26.328340+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:41:26.328340+0000","last_undegraded":"2024-04-22T08:41:26.328340+0000","last_fullsized":"2024-04-22T08:41:26.328340+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20168","last_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_deep_scrub":"41'20168","last_deep_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.328296+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:46:53.686238+0000","scrub_duration":0.019265356000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20386,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:20.865130+0000","last_change":"2024-04-22T08:41:20.865833+0000","last_active":"2024-04-22T08:41:20.865130+0000","last_peered":"2024-04-22T08:41:20.865130+0000","last_clean":"2024-04-22T08:41:20.865130+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:41:20.865130+0000","last_undegraded":"2024-04-22T08:41:20.865130+0000","last_fullsized":"2024-04-22T08:41:20.865130+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20232","last_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_deep_scrub":"41'20232","last_deep_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_clean_scrub_stamp":"2024-04-22T08:41:20.865071+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:16.784176+0000","scrub_duration":0.072564135000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20427,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731762+0000","last_change":"2024-04-22T08:38:18.712321+0000","last_active":"2024-04-22T08:40:58.731762+0000","last_peered":"2024-04-22T08:40:58.731762+0000","last_clean":"2024-04-22T08:40:58.731762+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:40:58.731762+0000","last_undegraded":"2024-04-22T08:40:58.731762+0000","last_fullsized":"2024-04-22T08:40:58.731762+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3232","last_scrub_stamp":"2024-04-22T08:27:44.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:44.400987+0000","objects_scrubbed":3169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.5564737609999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20183,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:19.839895+0000","last_change":"2024-04-22T08:41:19.840157+0000","last_active":"2024-04-22T08:41:19.839895+0000","last_peered":"2024-04-22T08:41:19.839895+0000","last_clean":"2024-04-22T08:41:19.839895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:41:19.839895+0000","last_undegraded":"2024-04-22T08:41:19.839895+0000","last_fullsized":"2024-04-22T08:41:19.839895+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20114","last_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_deep_scrub":"41'20114","last_deep_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_clean_scrub_stamp":"2024-04-22T08:41:19.839837+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:17:53.687937+0000","scrub_duration":0.056880157000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20096,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:23.372008+0000","last_change":"2024-04-22T08:41:23.372250+0000","last_active":"2024-04-22T08:41:23.372008+0000","last_peered":"2024-04-22T08:41:23.372008+0000","last_clean":"2024-04-22T08:41:23.372008+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:41:23.372008+0000","last_undegraded":"2024-04-22T08:41:23.372008+0000","last_fullsized":"2024-04-22T08:41:23.372008+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19919","last_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_deep_scrub":"41'19919","last_deep_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_clean_scrub_stamp":"2024-04-22T08:41:23.371965+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:52.833732+0000","scrub_duration":0.060232419000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28010,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731688+0000","last_change":"2024-04-22T08:35:02.647930+0000","last_active":"2024-04-22T08:40:58.731688+0000","last_peered":"2024-04-22T08:40:58.731688+0000","last_clean":"2024-04-22T08:40:58.731688+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:40:58.731688+0000","last_undegraded":"2024-04-22T08:40:58.731688+0000","last_fullsized":"2024-04-22T08:40:58.731688+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'3186","last_scrub_stamp":"2024-04-22T08:27:42.225197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:42.225197+0000","objects_scrubbed":3126,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.6537587440000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27628,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:07.598885+0000","last_change":"2024-04-22T08:30:10.170014+0000","last_active":"2024-04-22T08:41:07.598885+0000","last_peered":"2024-04-22T08:41:07.598885+0000","last_clean":"2024-04-22T08:41:07.598885+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:41:07.598885+0000","last_undegraded":"2024-04-22T08:41:07.598885+0000","last_fullsized":"2024-04-22T08:41:07.598885+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3707","last_scrub_stamp":"2024-04-22T08:27:52.930922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:52.930922+0000","objects_scrubbed":3626,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for deep scrub","scrub_duration":2.3055966950000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19989,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:15.375296+0000","last_change":"2024-04-22T08:41:15.375502+0000","last_active":"2024-04-22T08:41:15.375296+0000","last_peered":"2024-04-22T08:41:15.375296+0000","last_clean":"2024-04-22T08:41:15.375296+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:41:15.375296+0000","last_undegraded":"2024-04-22T08:41:15.375296+0000","last_fullsized":"2024-04-22T08:41:15.375296+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19806","last_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_deep_scrub":"41'19806","last_deep_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_clean_scrub_stamp":"2024-04-22T08:41:15.375259+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:09.919650+0000","scrub_duration":0.067500488999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:11.996871+0000","last_change":"2024-04-22T08:41:11.997115+0000","last_active":"2024-04-22T08:41:11.996871+0000","last_peered":"2024-04-22T08:41:11.996871+0000","last_clean":"2024-04-22T08:41:11.996871+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:41:11.996871+0000","last_undegraded":"2024-04-22T08:41:11.996871+0000","last_fullsized":"2024-04-22T08:41:11.996871+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_clean_scrub_stamp":"2024-04-22T08:41:11.996822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:55.505253+0000","scrub_duration":0.0048091319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20714,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:05.176670+0000","last_change":"2024-04-22T08:41:05.176670+0000","last_active":"2024-04-22T08:41:05.176670+0000","last_peered":"2024-04-22T08:41:05.176670+0000","last_clean":"2024-04-22T08:41:05.176670+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:41:05.176670+0000","last_undegraded":"2024-04-22T08:41:05.176670+0000","last_fullsized":"2024-04-22T08:41:05.176670+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20221","last_scrub_stamp":"2024-04-22T08:41:05.176408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:05.176408+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.041926848000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:14.817045+0000","last_change":"2024-04-22T08:41:14.817266+0000","last_active":"2024-04-22T08:41:14.817045+0000","last_peered":"2024-04-22T08:41:14.817045+0000","last_clean":"2024-04-22T08:41:14.817045+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:41:14.817045+0000","last_undegraded":"2024-04-22T08:41:14.817045+0000","last_fullsized":"2024-04-22T08:41:14.817045+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_clean_scrub_stamp":"2024-04-22T08:41:14.816997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:36:01.752249+0000","scrub_duration":0.038557098999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27871,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:24.411534+0000","last_change":"2024-04-22T08:41:24.411534+0000","last_active":"2024-04-22T08:41:24.411534+0000","last_peered":"2024-04-22T08:41:24.411534+0000","last_clean":"2024-04-22T08:41:24.411534+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:41:24.411534+0000","last_undegraded":"2024-04-22T08:41:24.411534+0000","last_fullsized":"2024-04-22T08:41:24.411534+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20186","last_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_deep_scrub":"41'20186","last_deep_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_clean_scrub_stamp":"2024-04-22T08:41:24.411274+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:03.386662+0000","scrub_duration":0.067090497999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":107,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:22.327379+0000","last_change":"2024-04-22T08:41:22.327379+0000","last_active":"2024-04-22T08:41:22.327379+0000","last_peered":"2024-04-22T08:41:22.327379+0000","last_clean":"2024-04-22T08:41:22.327379+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:41:22.327379+0000","last_undegraded":"2024-04-22T08:41:22.327379+0000","last_fullsized":"2024-04-22T08:41:22.327379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_clean_scrub_stamp":"2024-04-22T08:41:22.327076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:33.099040+0000","scrub_duration":0.047590474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27628,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:12.821685+0000","last_change":"2024-04-22T08:41:12.821975+0000","last_active":"2024-04-22T08:41:12.821685+0000","last_peered":"2024-04-22T08:41:12.821685+0000","last_clean":"2024-04-22T08:41:12.821685+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:41:12.821685+0000","last_undegraded":"2024-04-22T08:41:12.821685+0000","last_fullsized":"2024-04-22T08:41:12.821685+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19929","last_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_deep_scrub":"41'19929","last_deep_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_clean_scrub_stamp":"2024-04-22T08:41:12.821631+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:57:23.256574+0000","scrub_duration":0.045123838999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:10.993465+0000","last_change":"2024-04-22T08:41:10.993573+0000","last_active":"2024-04-22T08:41:10.993465+0000","last_peered":"2024-04-22T08:41:10.993465+0000","last_clean":"2024-04-22T08:41:10.993465+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:41:10.993465+0000","last_undegraded":"2024-04-22T08:41:10.993465+0000","last_fullsized":"2024-04-22T08:41:10.993465+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_clean_scrub_stamp":"2024-04-22T08:41:10.993446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:22.774565+0000","scrub_duration":0.019827966999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27691,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:25.785910+0000","last_change":"2024-04-22T08:41:25.786129+0000","last_active":"2024-04-22T08:41:25.785910+0000","last_peered":"2024-04-22T08:41:25.785910+0000","last_clean":"2024-04-22T08:41:25.785910+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:25.785910+0000","last_undegraded":"2024-04-22T08:41:25.785910+0000","last_fullsized":"2024-04-22T08:41:25.785910+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_deep_scrub":"41'19934","last_deep_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_clean_scrub_stamp":"2024-04-22T08:41:25.785858+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:18:56.744898+0000","scrub_duration":0.058784598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20915,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731980+0000","last_change":"2024-04-22T08:39:48.973847+0000","last_active":"2024-04-22T08:40:58.731980+0000","last_peered":"2024-04-22T08:40:58.731980+0000","last_clean":"2024-04-22T08:40:58.731980+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:40:58.731980+0000","last_undegraded":"2024-04-22T08:40:58.731980+0000","last_fullsized":"2024-04-22T08:40:58.731980+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:26:27.267649+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20141,"num_write_kb":40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3724,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:18.002600+0000","last_change":"2024-04-22T08:41:18.002798+0000","last_active":"2024-04-22T08:41:18.002600+0000","last_peered":"2024-04-22T08:41:18.002600+0000","last_clean":"2024-04-22T08:41:18.002600+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:41:18.002600+0000","last_undegraded":"2024-04-22T08:41:18.002600+0000","last_fullsized":"2024-04-22T08:41:18.002600+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_clean_scrub_stamp":"2024-04-22T08:41:18.002550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:26.665118+0000","scrub_duration":0.0087780029999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20443,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731948+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:58.731948+0000","last_peered":"2024-04-22T08:40:58.731948+0000","last_clean":"2024-04-22T08:40:58.731948+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:58.731948+0000","last_undegraded":"2024-04-22T08:40:58.731948+0000","last_fullsized":"2024-04-22T08:40:58.731948+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for deep scrub","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:21.819174+0000","last_change":"2024-04-22T08:41:21.819490+0000","last_active":"2024-04-22T08:41:21.819174+0000","last_peered":"2024-04-22T08:41:21.819174+0000","last_clean":"2024-04-22T08:41:21.819174+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:41:21.819174+0000","last_undegraded":"2024-04-22T08:41:21.819174+0000","last_fullsized":"2024-04-22T08:41:21.819174+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_clean_scrub_stamp":"2024-04-22T08:41:21.819114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:52:32.713418+0000","scrub_duration":0.050105947999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:59.300874+0000","last_change":"2024-04-22T08:40:59.300874+0000","last_active":"2024-04-22T08:40:59.300874+0000","last_peered":"2024-04-22T08:40:59.300874+0000","last_clean":"2024-04-22T08:40:59.300874+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:40:59.300874+0000","last_undegraded":"2024-04-22T08:40:59.300874+0000","last_fullsized":"2024-04-22T08:40:59.300874+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:40:59.300717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:40:59.300717+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.061123362000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:13.987251+0000","last_change":"2024-04-22T08:41:13.987493+0000","last_active":"2024-04-22T08:41:13.987251+0000","last_peered":"2024-04-22T08:41:13.987251+0000","last_clean":"2024-04-22T08:41:13.987251+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:41:13.987251+0000","last_undegraded":"2024-04-22T08:41:13.987251+0000","last_fullsized":"2024-04-22T08:41:13.987251+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_clean_scrub_stamp":"2024-04-22T08:41:13.987199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:57:16.122867+0000","scrub_duration":0.046424799000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.078271+0000","last_change":"2024-04-22T08:41:26.078439+0000","last_active":"2024-04-22T08:41:26.078271+0000","last_peered":"2024-04-22T08:41:26.078271+0000","last_clean":"2024-04-22T08:41:26.078271+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:41:26.078271+0000","last_undegraded":"2024-04-22T08:41:26.078271+0000","last_fullsized":"2024-04-22T08:41:26.078271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.078235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:28.495349+0000","scrub_duration":0.064540178000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'20002","reported_seq":20755,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:04.194624+0000","last_change":"2024-04-22T08:41:04.194624+0000","last_active":"2024-04-22T08:41:04.194624+0000","last_peered":"2024-04-22T08:41:04.194624+0000","last_clean":"2024-04-22T08:41:04.194624+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:41:04.194624+0000","last_undegraded":"2024-04-22T08:41:04.194624+0000","last_fullsized":"2024-04-22T08:41:04.194624+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20002","last_scrub_stamp":"2024-04-22T08:41:04.194401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:41:04.194401+0000","objects_scrubbed":1,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.077576006000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193379,"num_write_kb":333930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26531,"num_bytes_recovered":23019520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34119680,"data_stored":34112722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160616,"ondisk_log_size":160616,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117169,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395336,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93976504,"statfs":{"total":96636764160,"available":96231940096,"internally_reserved":0,"allocated":35586048,"data_stored":35260472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.70299999999999996,"15min":0.94899999999999995},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.53200000000000003,"5min":6.3860000000000001,"15min":13.635},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.79300000000000004,"15min":0.81200000000000006},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.69899999999999995,"5min":6.431,"15min":13.721},"last":0.67600000000000005}]},{"osd":1,"last update":"Mon Apr 22 08:41:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47199999999999998,"15min":1.0089999999999999},"min":{"1min":0.254,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":1.419,"15min":15.6},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.76000000000000001,"15min":1.0289999999999999},"min":{"1min":0.27900000000000003,"5min":0.23400000000000001,"15min":0.20499999999999999},"max":{"1min":0.72799999999999998,"5min":6.6239999999999997,"15min":18.32},"last":0.63}]}]},{"osd":1,"up_from":33,"seq":141733920931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56699999999999995,"15min":0.755},"min":{"1min":0.42299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.84399999999999997,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.621,"15min":0.85599999999999998},"min":{"1min":0.57599999999999996,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":0.95799999999999996,"5min":4.2439999999999998,"15min":6.8860000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.49199999999999999,"15min":1.099},"min":{"1min":0.44900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.78100000000000003,"5min":3.5950000000000002,"15min":15.989000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":1.0589999999999999},"min":{"1min":0.442,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.76300000000000001,"5min":3.625,"15min":20.149000000000001},"last":0.60799999999999998}]}]},{"osd":0,"up_from":21,"seq":90194313399,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:41:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.68500000000000005,"15min":0.99299999999999999},"min":{"1min":0.192,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.70899999999999996,"5min":10.435,"15min":11.368},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.91900000000000004},"min":{"1min":0.216,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.751,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Apr 22 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.86699999999999999,"15min":0.94299999999999995},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.18099999999999999},"max":{"1min":1.014,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.2589999999999999,"15min":1.0529999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":1.2110000000000001,"5min":11.141999999999999,"15min":12.131},"last":0.70399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34111488,"data_stored":34111488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:31.891 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-22T08:27:44.400987+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=44, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.891 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-22T08:27:42.225197+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=42, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.891 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-22T08:27:52.930922+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=52, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.891 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-22T08:41:05.176408+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=5, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.891 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-22T08:26:27.267649+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=26, tm_sec=27, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.892 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-22T08:27:53.920523+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=53, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.892 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-22T08:40:59.300717+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=40, tm_sec=59, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.892 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-22T08:41:04.194401+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=4, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:31.892 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T08:41:51.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:41:52.202 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:41:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:41:52.220 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-04-22T08:41:50.507486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173238,"num_write_kb":293649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":23015424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160617,"ondisk_log_size":160617,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1358284,"kb_used_data":19828,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281757236,"statfs":{"total":289910292480,"available":288519409664,"internally_reserved":0,"allocated":20303872,"data_stored":19326648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-617,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-20141,"num_write_kb":-40281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3724,"num_bytes_recovered":-4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27868,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.328340+0000","last_change":"2024-04-22T08:41:26.328540+0000","last_active":"2024-04-22T08:41:26.328340+0000","last_peered":"2024-04-22T08:41:26.328340+0000","last_clean":"2024-04-22T08:41:26.328340+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:41:26.328340+0000","last_undegraded":"2024-04-22T08:41:26.328340+0000","last_fullsized":"2024-04-22T08:41:26.328340+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20168","last_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_deep_scrub":"41'20168","last_deep_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.328296+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:46:53.686238+0000","scrub_duration":0.019265356000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20386,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:20.865130+0000","last_change":"2024-04-22T08:41:20.865833+0000","last_active":"2024-04-22T08:41:20.865130+0000","last_peered":"2024-04-22T08:41:20.865130+0000","last_clean":"2024-04-22T08:41:20.865130+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:41:20.865130+0000","last_undegraded":"2024-04-22T08:41:20.865130+0000","last_fullsized":"2024-04-22T08:41:20.865130+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20232","last_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_deep_scrub":"41'20232","last_deep_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_clean_scrub_stamp":"2024-04-22T08:41:20.865071+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:16.784176+0000","scrub_duration":0.072564135000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20438,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:31.122544+0000","last_change":"2024-04-22T08:41:31.122769+0000","last_active":"2024-04-22T08:41:31.122544+0000","last_peered":"2024-04-22T08:41:31.122544+0000","last_clean":"2024-04-22T08:41:31.122544+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:41:31.122544+0000","last_undegraded":"2024-04-22T08:41:31.122544+0000","last_fullsized":"2024-04-22T08:41:31.122544+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19864","last_scrub_stamp":"2024-04-22T08:41:31.122491+0000","last_deep_scrub":"41'19864","last_deep_scrub_stamp":"2024-04-22T08:41:31.122491+0000","last_clean_scrub_stamp":"2024-04-22T08:41:31.122491+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:06:22.846194+0000","scrub_duration":0.053281639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20183,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:19.839895+0000","last_change":"2024-04-22T08:41:19.840157+0000","last_active":"2024-04-22T08:41:19.839895+0000","last_peered":"2024-04-22T08:41:19.839895+0000","last_clean":"2024-04-22T08:41:19.839895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:41:19.839895+0000","last_undegraded":"2024-04-22T08:41:19.839895+0000","last_fullsized":"2024-04-22T08:41:19.839895+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20114","last_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_deep_scrub":"41'20114","last_deep_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_clean_scrub_stamp":"2024-04-22T08:41:19.839837+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:17:53.687937+0000","scrub_duration":0.056880157000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20096,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:23.372008+0000","last_change":"2024-04-22T08:41:23.372250+0000","last_active":"2024-04-22T08:41:23.372008+0000","last_peered":"2024-04-22T08:41:23.372008+0000","last_clean":"2024-04-22T08:41:23.372008+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:41:23.372008+0000","last_undegraded":"2024-04-22T08:41:23.372008+0000","last_fullsized":"2024-04-22T08:41:23.372008+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19919","last_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_deep_scrub":"41'19919","last_deep_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_clean_scrub_stamp":"2024-04-22T08:41:23.371965+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:52.833732+0000","scrub_duration":0.060232419000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28022,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:29.144575+0000","last_change":"2024-04-22T08:41:29.144575+0000","last_active":"2024-04-22T08:41:29.144575+0000","last_peered":"2024-04-22T08:41:29.144575+0000","last_clean":"2024-04-22T08:41:29.144575+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:41:29.144575+0000","last_undegraded":"2024-04-22T08:41:29.144575+0000","last_fullsized":"2024-04-22T08:41:29.144575+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20047","last_scrub_stamp":"2024-04-22T08:41:29.144322+0000","last_deep_scrub":"41'20047","last_deep_scrub_stamp":"2024-04-22T08:41:29.144322+0000","last_clean_scrub_stamp":"2024-04-22T08:41:29.144322+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:26:07.141985+0000","scrub_duration":0.078157180000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:28.151674+0000","last_change":"2024-04-22T08:41:28.151901+0000","last_active":"2024-04-22T08:41:28.151674+0000","last_peered":"2024-04-22T08:41:28.151674+0000","last_clean":"2024-04-22T08:41:28.151674+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:41:28.151674+0000","last_undegraded":"2024-04-22T08:41:28.151674+0000","last_fullsized":"2024-04-22T08:41:28.151674+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19864","last_scrub_stamp":"2024-04-22T08:41:28.151619+0000","last_deep_scrub":"41'19864","last_deep_scrub_stamp":"2024-04-22T08:41:28.151619+0000","last_clean_scrub_stamp":"2024-04-22T08:41:28.151619+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:27.751792+0000","scrub_duration":0.074865033999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19989,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:15.375296+0000","last_change":"2024-04-22T08:41:15.375502+0000","last_active":"2024-04-22T08:41:15.375296+0000","last_peered":"2024-04-22T08:41:15.375296+0000","last_clean":"2024-04-22T08:41:15.375296+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:41:15.375296+0000","last_undegraded":"2024-04-22T08:41:15.375296+0000","last_fullsized":"2024-04-22T08:41:15.375296+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19806","last_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_deep_scrub":"41'19806","last_deep_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_clean_scrub_stamp":"2024-04-22T08:41:15.375259+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:09.919650+0000","scrub_duration":0.067500488999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:11.996871+0000","last_change":"2024-04-22T08:41:11.997115+0000","last_active":"2024-04-22T08:41:11.996871+0000","last_peered":"2024-04-22T08:41:11.996871+0000","last_clean":"2024-04-22T08:41:11.996871+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:41:11.996871+0000","last_undegraded":"2024-04-22T08:41:11.996871+0000","last_fullsized":"2024-04-22T08:41:11.996871+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_clean_scrub_stamp":"2024-04-22T08:41:11.996822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:55.505253+0000","scrub_duration":0.0048091319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20726,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:35.193193+0000","last_change":"2024-04-22T08:41:35.193193+0000","last_active":"2024-04-22T08:41:35.193193+0000","last_peered":"2024-04-22T08:41:35.193193+0000","last_clean":"2024-04-22T08:41:35.193193+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:41:35.193193+0000","last_undegraded":"2024-04-22T08:41:35.193193+0000","last_fullsized":"2024-04-22T08:41:35.193193+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20221","last_scrub_stamp":"2024-04-22T08:41:35.192911+0000","last_deep_scrub":"41'20221","last_deep_scrub_stamp":"2024-04-22T08:41:35.192911+0000","last_clean_scrub_stamp":"2024-04-22T08:41:35.192911+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:25.592371+0000","scrub_duration":0.042583906999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:14.817266+0000","last_change":"2024-04-22T08:41:14.817266+0000","last_active":"2024-04-22T08:41:14.817266+0000","last_peered":"2024-04-22T08:41:14.817266+0000","last_clean":"2024-04-22T08:41:14.817266+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:41:14.817266+0000","last_undegraded":"2024-04-22T08:41:14.817266+0000","last_fullsized":"2024-04-22T08:41:14.817266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_clean_scrub_stamp":"2024-04-22T08:41:14.816997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:36:01.752249+0000","scrub_duration":0.038557098999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27871,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:24.411534+0000","last_change":"2024-04-22T08:41:24.411534+0000","last_active":"2024-04-22T08:41:24.411534+0000","last_peered":"2024-04-22T08:41:24.411534+0000","last_clean":"2024-04-22T08:41:24.411534+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:41:24.411534+0000","last_undegraded":"2024-04-22T08:41:24.411534+0000","last_fullsized":"2024-04-22T08:41:24.411534+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20186","last_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_deep_scrub":"41'20186","last_deep_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_clean_scrub_stamp":"2024-04-22T08:41:24.411274+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:03.386662+0000","scrub_duration":0.067090497999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":107,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:22.327379+0000","last_change":"2024-04-22T08:41:22.327379+0000","last_active":"2024-04-22T08:41:22.327379+0000","last_peered":"2024-04-22T08:41:22.327379+0000","last_clean":"2024-04-22T08:41:22.327379+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:41:22.327379+0000","last_undegraded":"2024-04-22T08:41:22.327379+0000","last_fullsized":"2024-04-22T08:41:22.327379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_clean_scrub_stamp":"2024-04-22T08:41:22.327076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:33.099040+0000","scrub_duration":0.047590474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27629,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:12.821975+0000","last_change":"2024-04-22T08:41:12.821975+0000","last_active":"2024-04-22T08:41:12.821975+0000","last_peered":"2024-04-22T08:41:12.821975+0000","last_clean":"2024-04-22T08:41:12.821975+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:41:12.821975+0000","last_undegraded":"2024-04-22T08:41:12.821975+0000","last_fullsized":"2024-04-22T08:41:12.821975+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19929","last_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_deep_scrub":"41'19929","last_deep_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_clean_scrub_stamp":"2024-04-22T08:41:12.821631+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:57:23.256574+0000","scrub_duration":0.045123838999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:10.993465+0000","last_change":"2024-04-22T08:41:10.993573+0000","last_active":"2024-04-22T08:41:10.993465+0000","last_peered":"2024-04-22T08:41:10.993465+0000","last_clean":"2024-04-22T08:41:10.993465+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:41:10.993465+0000","last_undegraded":"2024-04-22T08:41:10.993465+0000","last_fullsized":"2024-04-22T08:41:10.993465+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_clean_scrub_stamp":"2024-04-22T08:41:10.993446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:22.774565+0000","scrub_duration":0.019827966999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27691,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:25.785910+0000","last_change":"2024-04-22T08:41:25.786129+0000","last_active":"2024-04-22T08:41:25.785910+0000","last_peered":"2024-04-22T08:41:25.785910+0000","last_clean":"2024-04-22T08:41:25.785910+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:25.785910+0000","last_undegraded":"2024-04-22T08:41:25.785910+0000","last_fullsized":"2024-04-22T08:41:25.785910+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_deep_scrub":"41'19934","last_deep_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_clean_scrub_stamp":"2024-04-22T08:41:25.785858+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:18:56.744898+0000","scrub_duration":0.058784598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20927,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:43.054944+0000","last_change":"2024-04-22T08:41:43.055218+0000","last_active":"2024-04-22T08:41:43.054944+0000","last_peered":"2024-04-22T08:41:43.054944+0000","last_clean":"2024-04-22T08:41:43.054944+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:41:43.054944+0000","last_undegraded":"2024-04-22T08:41:43.054944+0000","last_fullsized":"2024-04-22T08:41:43.054944+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20142","last_scrub_stamp":"2024-04-22T08:41:43.054893+0000","last_deep_scrub":"41'20142","last_deep_scrub_stamp":"2024-04-22T08:41:43.054893+0000","last_clean_scrub_stamp":"2024-04-22T08:41:43.054893+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:45.020226+0000","scrub_duration":0.037486851000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:18.002600+0000","last_change":"2024-04-22T08:41:18.002798+0000","last_active":"2024-04-22T08:41:18.002600+0000","last_peered":"2024-04-22T08:41:18.002600+0000","last_clean":"2024-04-22T08:41:18.002600+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:41:18.002600+0000","last_undegraded":"2024-04-22T08:41:18.002600+0000","last_fullsized":"2024-04-22T08:41:18.002600+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_clean_scrub_stamp":"2024-04-22T08:41:18.002550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:26.665118+0000","scrub_duration":0.0087780029999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20443,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:40:58.731948+0000","last_change":"2024-04-22T08:40:06.332789+0000","last_active":"2024-04-22T08:40:58.731948+0000","last_peered":"2024-04-22T08:40:58.731948+0000","last_clean":"2024-04-22T08:40:58.731948+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:40:58.731948+0000","last_undegraded":"2024-04-22T08:40:58.731948+0000","last_fullsized":"2024-04-22T08:40:58.731948+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'3834","last_scrub_stamp":"2024-04-22T08:27:53.920523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:26:27.267649+0000","last_clean_scrub_stamp":"2024-04-22T08:27:53.920523+0000","objects_scrubbed":3761,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for deep scrub","scrub_duration":2.2916779370000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":71,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:21.819490+0000","last_change":"2024-04-22T08:41:21.819490+0000","last_active":"2024-04-22T08:41:21.819490+0000","last_peered":"2024-04-22T08:41:21.819490+0000","last_clean":"2024-04-22T08:41:21.819490+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:41:21.819490+0000","last_undegraded":"2024-04-22T08:41:21.819490+0000","last_fullsized":"2024-04-22T08:41:21.819490+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_clean_scrub_stamp":"2024-04-22T08:41:21.819114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:52:32.713418+0000","scrub_duration":0.050105947999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20692,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:30.118155+0000","last_change":"2024-04-22T08:41:30.118394+0000","last_active":"2024-04-22T08:41:30.118155+0000","last_peered":"2024-04-22T08:41:30.118155+0000","last_clean":"2024-04-22T08:41:30.118155+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:41:30.118155+0000","last_undegraded":"2024-04-22T08:41:30.118155+0000","last_fullsized":"2024-04-22T08:41:30.118155+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:41:30.118102+0000","last_deep_scrub":"41'20038","last_deep_scrub_stamp":"2024-04-22T08:41:30.118102+0000","last_clean_scrub_stamp":"2024-04-22T08:41:30.118102+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:26:46.931464+0000","scrub_duration":0.066040161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:13.987251+0000","last_change":"2024-04-22T08:41:13.987493+0000","last_active":"2024-04-22T08:41:13.987251+0000","last_peered":"2024-04-22T08:41:13.987251+0000","last_clean":"2024-04-22T08:41:13.987251+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:41:13.987251+0000","last_undegraded":"2024-04-22T08:41:13.987251+0000","last_fullsized":"2024-04-22T08:41:13.987251+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_clean_scrub_stamp":"2024-04-22T08:41:13.987199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:57:16.122867+0000","scrub_duration":0.046424799000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.078271+0000","last_change":"2024-04-22T08:41:26.078439+0000","last_active":"2024-04-22T08:41:26.078271+0000","last_peered":"2024-04-22T08:41:26.078271+0000","last_clean":"2024-04-22T08:41:26.078271+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:41:26.078271+0000","last_undegraded":"2024-04-22T08:41:26.078271+0000","last_fullsized":"2024-04-22T08:41:26.078271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.078235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:28.495349+0000","scrub_duration":0.064540178000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"59'20003","reported_seq":20767,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:32.176811+0000","last_change":"2024-04-22T08:41:32.177034+0000","last_active":"2024-04-22T08:41:32.176811+0000","last_peered":"2024-04-22T08:41:32.176811+0000","last_clean":"2024-04-22T08:41:32.176811+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:41:32.176811+0000","last_undegraded":"2024-04-22T08:41:32.176811+0000","last_fullsized":"2024-04-22T08:41:32.176811+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'20003","last_scrub_stamp":"2024-04-22T08:41:32.176751+0000","last_deep_scrub":"59'20003","last_deep_scrub_stamp":"2024-04-22T08:41:32.176751+0000","last_clean_scrub_stamp":"2024-04-22T08:41:32.176751+0000","objects_scrubbed":1,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:36.014650+0000","scrub_duration":0.088016560999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173238,"num_write_kb":293649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":23015424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18837504,"data_stored":18830546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160617,"ondisk_log_size":160617,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379452,"kb_used_data":18868,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93992388,"statfs":{"total":96636764160,"available":96248205312,"internally_reserved":0,"allocated":19320832,"data_stored":18995256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.58199999999999996,"15min":0.93000000000000005},"min":{"1min":0.26400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.58799999999999997,"5min":6.3860000000000001,"15min":13.635},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.69199999999999995,"15min":0.81899999999999995},"min":{"1min":0.376,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.70099999999999996,"5min":6.431,"15min":13.721},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:41:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47199999999999998,"15min":1.0089999999999999},"min":{"1min":0.254,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":1.419,"15min":15.6},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.76000000000000001,"15min":1.0289999999999999},"min":{"1min":0.27900000000000003,"5min":0.23400000000000001,"15min":0.20499999999999999},"max":{"1min":0.72799999999999998,"5min":6.6239999999999997,"15min":18.32},"last":0.61499999999999999}]}]},{"osd":1,"up_from":33,"seq":141733920935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56699999999999995,"15min":0.755},"min":{"1min":0.42299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.84399999999999997,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.621,"15min":0.85599999999999998},"min":{"1min":0.57599999999999996,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":0.95799999999999996,"5min":4.2439999999999998,"15min":6.8860000000000001},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Apr 22 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.49199999999999999,"15min":1.099},"min":{"1min":0.44900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.78100000000000003,"5min":3.5950000000000002,"15min":15.989000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":1.0589999999999999},"min":{"1min":0.442,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.76300000000000001,"5min":3.625,"15min":20.149000000000001},"last":0.61199999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313403,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:41:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.68500000000000005,"15min":0.99299999999999999},"min":{"1min":0.192,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.70899999999999996,"5min":10.435,"15min":11.368},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.91900000000000004},"min":{"1min":0.216,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.751,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.75800000000000001,"15min":0.93200000000000005},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.18099999999999999},"max":{"1min":0.78300000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.0649999999999999,"15min":1.0449999999999999},"min":{"1min":0.39500000000000002,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.78500000000000003,"5min":11.141999999999999,"15min":12.131},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18829312,"data_stored":18829312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:41:52.222 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-22T08:27:53.920523+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=27, tm_sec=53, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=41, tm_sec=8, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:41:52.223 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T08:42:12.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T08:42:12.532 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T08:42:12.550 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-04-22T08:42:12.513278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173238,"num_write_kb":293649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":23015424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160617,"ondisk_log_size":160617,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1348556,"kb_used_data":10100,"kb_used_omap":12,"kb_used_meta":1338419,"kb_avail":281766964,"statfs":{"total":289910292480,"available":288529371136,"internally_reserved":0,"allocated":10342400,"data_stored":9365176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13202,"internal_metadata":1370541166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"2.f","version":"41'20168","reported_seq":27868,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.328340+0000","last_change":"2024-04-22T08:41:26.328540+0000","last_active":"2024-04-22T08:41:26.328340+0000","last_peered":"2024-04-22T08:41:26.328340+0000","last_clean":"2024-04-22T08:41:26.328340+0000","last_became_active":"2024-04-22T08:40:58.732628+0000","last_became_peered":"2024-04-22T08:40:58.732628+0000","last_unstale":"2024-04-22T08:41:26.328340+0000","last_undegraded":"2024-04-22T08:41:26.328340+0000","last_fullsized":"2024-04-22T08:41:26.328340+0000","mapping_epoch":58,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20168","last_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_deep_scrub":"41'20168","last_deep_scrub_stamp":"2024-04-22T08:41:26.328296+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.328296+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:46:53.686238+0000","scrub_duration":0.019265356000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20168,"num_write_kb":40336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'20232","reported_seq":20386,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:20.865130+0000","last_change":"2024-04-22T08:41:20.865833+0000","last_active":"2024-04-22T08:41:20.865130+0000","last_peered":"2024-04-22T08:41:20.865130+0000","last_clean":"2024-04-22T08:41:20.865130+0000","last_became_active":"2024-04-22T08:27:06.395613+0000","last_became_peered":"2024-04-22T08:27:06.395613+0000","last_unstale":"2024-04-22T08:41:20.865130+0000","last_undegraded":"2024-04-22T08:41:20.865130+0000","last_fullsized":"2024-04-22T08:41:20.865130+0000","mapping_epoch":27,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20232","last_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_deep_scrub":"41'20232","last_deep_scrub_stamp":"2024-04-22T08:41:20.865071+0000","last_clean_scrub_stamp":"2024-04-22T08:41:20.865071+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:16.784176+0000","scrub_duration":0.072564135000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":28700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'19864","reported_seq":20438,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:31.122544+0000","last_change":"2024-04-22T08:41:31.122769+0000","last_active":"2024-04-22T08:41:31.122544+0000","last_peered":"2024-04-22T08:41:31.122544+0000","last_clean":"2024-04-22T08:41:31.122544+0000","last_became_active":"2024-04-22T08:38:18.571999+0000","last_became_peered":"2024-04-22T08:38:18.571999+0000","last_unstale":"2024-04-22T08:41:31.122544+0000","last_undegraded":"2024-04-22T08:41:31.122544+0000","last_fullsized":"2024-04-22T08:41:31.122544+0000","mapping_epoch":46,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19864","last_scrub_stamp":"2024-04-22T08:41:31.122491+0000","last_deep_scrub":"41'19864","last_deep_scrub_stamp":"2024-04-22T08:41:31.122491+0000","last_clean_scrub_stamp":"2024-04-22T08:41:31.122491+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:06:22.846194+0000","scrub_duration":0.053281639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16691,"num_write_kb":27041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'20114","reported_seq":20183,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:19.839895+0000","last_change":"2024-04-22T08:41:19.840157+0000","last_active":"2024-04-22T08:41:19.839895+0000","last_peered":"2024-04-22T08:41:19.839895+0000","last_clean":"2024-04-22T08:41:19.839895+0000","last_became_active":"2024-04-22T08:27:06.397657+0000","last_became_peered":"2024-04-22T08:27:06.397657+0000","last_unstale":"2024-04-22T08:41:19.839895+0000","last_undegraded":"2024-04-22T08:41:19.839895+0000","last_fullsized":"2024-04-22T08:41:19.839895+0000","mapping_epoch":27,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20114","last_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_deep_scrub":"41'20114","last_deep_scrub_stamp":"2024-04-22T08:41:19.839837+0000","last_clean_scrub_stamp":"2024-04-22T08:41:19.839837+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:17:53.687937+0000","scrub_duration":0.056880157000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20114,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'19919","reported_seq":20096,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:23.372008+0000","last_change":"2024-04-22T08:41:23.372250+0000","last_active":"2024-04-22T08:41:23.372008+0000","last_peered":"2024-04-22T08:41:23.372008+0000","last_clean":"2024-04-22T08:41:23.372008+0000","last_became_active":"2024-04-22T08:27:06.397481+0000","last_became_peered":"2024-04-22T08:27:06.397481+0000","last_unstale":"2024-04-22T08:41:23.372008+0000","last_undegraded":"2024-04-22T08:41:23.372008+0000","last_fullsized":"2024-04-22T08:41:23.372008+0000","mapping_epoch":27,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19919","last_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_deep_scrub":"41'19919","last_deep_scrub_stamp":"2024-04-22T08:41:23.371965+0000","last_clean_scrub_stamp":"2024-04-22T08:41:23.371965+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:52.833732+0000","scrub_duration":0.060232419000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17011,"num_write_kb":28220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'20047","reported_seq":28022,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:29.144575+0000","last_change":"2024-04-22T08:41:29.144575+0000","last_active":"2024-04-22T08:41:29.144575+0000","last_peered":"2024-04-22T08:41:29.144575+0000","last_clean":"2024-04-22T08:41:29.144575+0000","last_became_active":"2024-04-22T08:35:02.582433+0000","last_became_peered":"2024-04-22T08:35:02.582433+0000","last_unstale":"2024-04-22T08:41:29.144575+0000","last_undegraded":"2024-04-22T08:41:29.144575+0000","last_fullsized":"2024-04-22T08:41:29.144575+0000","mapping_epoch":40,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20047","last_scrub_stamp":"2024-04-22T08:41:29.144322+0000","last_deep_scrub":"41'20047","last_deep_scrub_stamp":"2024-04-22T08:41:29.144322+0000","last_clean_scrub_stamp":"2024-04-22T08:41:29.144322+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:26:07.141985+0000","scrub_duration":0.078157180000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16916,"num_write_kb":27580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3882,"num_bytes_recovered":7528448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"41'19864","reported_seq":27639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:28.151674+0000","last_change":"2024-04-22T08:41:28.151901+0000","last_active":"2024-04-22T08:41:28.151674+0000","last_peered":"2024-04-22T08:41:28.151674+0000","last_clean":"2024-04-22T08:41:28.151674+0000","last_became_active":"2024-04-22T08:30:10.152874+0000","last_became_peered":"2024-04-22T08:30:10.152874+0000","last_unstale":"2024-04-22T08:41:28.151674+0000","last_undegraded":"2024-04-22T08:41:28.151674+0000","last_fullsized":"2024-04-22T08:41:28.151674+0000","mapping_epoch":36,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19864","last_scrub_stamp":"2024-04-22T08:41:28.151619+0000","last_deep_scrub":"41'19864","last_deep_scrub_stamp":"2024-04-22T08:41:28.151619+0000","last_clean_scrub_stamp":"2024-04-22T08:41:28.151619+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:27.751792+0000","scrub_duration":0.074865033999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3783,"num_bytes_recovered":15486976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'19806","reported_seq":19990,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:15.375502+0000","last_change":"2024-04-22T08:41:15.375502+0000","last_active":"2024-04-22T08:41:15.375502+0000","last_peered":"2024-04-22T08:41:15.375502+0000","last_clean":"2024-04-22T08:41:15.375502+0000","last_became_active":"2024-04-22T08:27:06.390170+0000","last_became_peered":"2024-04-22T08:27:06.390170+0000","last_unstale":"2024-04-22T08:41:15.375502+0000","last_undegraded":"2024-04-22T08:41:15.375502+0000","last_fullsized":"2024-04-22T08:41:15.375502+0000","mapping_epoch":27,"log_start":"37'9800","ondisk_log_start":"37'9800","created":12,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19806","last_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_deep_scrub":"41'19806","last_deep_scrub_stamp":"2024-04-22T08:41:15.375259+0000","last_clean_scrub_stamp":"2024-04-22T08:41:15.375259+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:09.919650+0000","scrub_duration":0.067500488999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16454,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:11.997115+0000","last_change":"2024-04-22T08:41:11.997115+0000","last_active":"2024-04-22T08:41:11.997115+0000","last_peered":"2024-04-22T08:41:11.997115+0000","last_clean":"2024-04-22T08:41:11.997115+0000","last_became_active":"2024-04-22T08:27:56.106178+0000","last_became_peered":"2024-04-22T08:27:56.106178+0000","last_unstale":"2024-04-22T08:41:11.997115+0000","last_undegraded":"2024-04-22T08:41:11.997115+0000","last_fullsized":"2024-04-22T08:41:11.997115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:11.996822+0000","last_clean_scrub_stamp":"2024-04-22T08:41:11.996822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:55.505253+0000","scrub_duration":0.0048091319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'20221","reported_seq":20726,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:35.193193+0000","last_change":"2024-04-22T08:41:35.193193+0000","last_active":"2024-04-22T08:41:35.193193+0000","last_peered":"2024-04-22T08:41:35.193193+0000","last_clean":"2024-04-22T08:41:35.193193+0000","last_became_active":"2024-04-22T08:39:32.743109+0000","last_became_peered":"2024-04-22T08:39:32.743109+0000","last_unstale":"2024-04-22T08:41:35.193193+0000","last_undegraded":"2024-04-22T08:41:35.193193+0000","last_fullsized":"2024-04-22T08:41:35.193193+0000","mapping_epoch":50,"log_start":"37'10200","ondisk_log_start":"37'10200","created":12,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20221","last_scrub_stamp":"2024-04-22T08:41:35.192911+0000","last_deep_scrub":"41'20221","last_deep_scrub_stamp":"2024-04-22T08:41:35.192911+0000","last_clean_scrub_stamp":"2024-04-22T08:41:35.192911+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:25.592371+0000","scrub_duration":0.042583906999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:14.817266+0000","last_change":"2024-04-22T08:41:14.817266+0000","last_active":"2024-04-22T08:41:14.817266+0000","last_peered":"2024-04-22T08:41:14.817266+0000","last_clean":"2024-04-22T08:41:14.817266+0000","last_became_active":"2024-04-22T08:27:06.395601+0000","last_became_peered":"2024-04-22T08:27:06.395601+0000","last_unstale":"2024-04-22T08:41:14.817266+0000","last_undegraded":"2024-04-22T08:41:14.817266+0000","last_fullsized":"2024-04-22T08:41:14.817266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:14.816997+0000","last_clean_scrub_stamp":"2024-04-22T08:41:14.816997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:36:01.752249+0000","scrub_duration":0.038557098999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'20186","reported_seq":27871,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:24.411534+0000","last_change":"2024-04-22T08:41:24.411534+0000","last_active":"2024-04-22T08:41:24.411534+0000","last_peered":"2024-04-22T08:41:24.411534+0000","last_clean":"2024-04-22T08:41:24.411534+0000","last_became_active":"2024-04-22T08:40:21.578345+0000","last_became_peered":"2024-04-22T08:40:21.578345+0000","last_unstale":"2024-04-22T08:41:24.411534+0000","last_undegraded":"2024-04-22T08:41:24.411534+0000","last_fullsized":"2024-04-22T08:41:24.411534+0000","mapping_epoch":56,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20186","last_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_deep_scrub":"41'20186","last_deep_scrub_stamp":"2024-04-22T08:41:24.411274+0000","last_clean_scrub_stamp":"2024-04-22T08:41:24.411274+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:03.386662+0000","scrub_duration":0.067090497999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":107,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:22.327379+0000","last_change":"2024-04-22T08:41:22.327379+0000","last_active":"2024-04-22T08:41:22.327379+0000","last_peered":"2024-04-22T08:41:22.327379+0000","last_clean":"2024-04-22T08:41:22.327379+0000","last_became_active":"2024-04-22T08:27:56.249615+0000","last_became_peered":"2024-04-22T08:27:56.249615+0000","last_unstale":"2024-04-22T08:41:22.327379+0000","last_undegraded":"2024-04-22T08:41:22.327379+0000","last_fullsized":"2024-04-22T08:41:22.327379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:22.327076+0000","last_clean_scrub_stamp":"2024-04-22T08:41:22.327076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:25:33.099040+0000","scrub_duration":0.047590474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'19929","reported_seq":27629,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:12.821975+0000","last_change":"2024-04-22T08:41:12.821975+0000","last_active":"2024-04-22T08:41:12.821975+0000","last_peered":"2024-04-22T08:41:12.821975+0000","last_clean":"2024-04-22T08:41:12.821975+0000","last_became_active":"2024-04-22T08:37:18.604543+0000","last_became_peered":"2024-04-22T08:37:18.604543+0000","last_unstale":"2024-04-22T08:41:12.821975+0000","last_undegraded":"2024-04-22T08:41:12.821975+0000","last_fullsized":"2024-04-22T08:41:12.821975+0000","mapping_epoch":42,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19929","last_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_deep_scrub":"41'19929","last_deep_scrub_stamp":"2024-04-22T08:41:12.821631+0000","last_clean_scrub_stamp":"2024-04-22T08:41:12.821631+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:57:23.256574+0000","scrub_duration":0.045123838999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16383,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:10.993465+0000","last_change":"2024-04-22T08:41:10.993573+0000","last_active":"2024-04-22T08:41:10.993465+0000","last_peered":"2024-04-22T08:41:10.993465+0000","last_clean":"2024-04-22T08:41:10.993465+0000","last_became_active":"2024-04-22T08:27:56.216663+0000","last_became_peered":"2024-04-22T08:27:56.216663+0000","last_unstale":"2024-04-22T08:41:10.993465+0000","last_undegraded":"2024-04-22T08:41:10.993465+0000","last_fullsized":"2024-04-22T08:41:10.993465+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:10.993446+0000","last_clean_scrub_stamp":"2024-04-22T08:41:10.993446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:13:22.774565+0000","scrub_duration":0.019827966999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'19934","reported_seq":27691,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:25.785910+0000","last_change":"2024-04-22T08:41:25.786129+0000","last_active":"2024-04-22T08:41:25.785910+0000","last_peered":"2024-04-22T08:41:25.785910+0000","last_clean":"2024-04-22T08:41:25.785910+0000","last_became_active":"2024-04-22T08:32:19.148667+0000","last_became_peered":"2024-04-22T08:32:19.148667+0000","last_unstale":"2024-04-22T08:41:25.785910+0000","last_undegraded":"2024-04-22T08:41:25.785910+0000","last_fullsized":"2024-04-22T08:41:25.785910+0000","mapping_epoch":38,"log_start":"37'9900","ondisk_log_start":"37'9900","created":12,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19934","last_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_deep_scrub":"41'19934","last_deep_scrub_stamp":"2024-04-22T08:41:25.785858+0000","last_clean_scrub_stamp":"2024-04-22T08:41:25.785858+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:18:56.744898+0000","scrub_duration":0.058784598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'20142","reported_seq":20928,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:43.055218+0000","last_change":"2024-04-22T08:41:43.055218+0000","last_active":"2024-04-22T08:41:43.055218+0000","last_peered":"2024-04-22T08:41:43.055218+0000","last_clean":"2024-04-22T08:41:43.055218+0000","last_became_active":"2024-04-22T08:39:48.821746+0000","last_became_peered":"2024-04-22T08:39:48.821746+0000","last_unstale":"2024-04-22T08:41:43.055218+0000","last_undegraded":"2024-04-22T08:41:43.055218+0000","last_fullsized":"2024-04-22T08:41:43.055218+0000","mapping_epoch":52,"log_start":"37'10100","ondisk_log_start":"37'10100","created":12,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20142","last_scrub_stamp":"2024-04-22T08:41:43.054893+0000","last_deep_scrub":"41'20142","last_deep_scrub_stamp":"2024-04-22T08:41:43.054893+0000","last_clean_scrub_stamp":"2024-04-22T08:41:43.054893+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:45.020226+0000","scrub_duration":0.037486851000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:18.002600+0000","last_change":"2024-04-22T08:41:18.002798+0000","last_active":"2024-04-22T08:41:18.002600+0000","last_peered":"2024-04-22T08:41:18.002600+0000","last_clean":"2024-04-22T08:41:18.002600+0000","last_became_active":"2024-04-22T08:27:56.351763+0000","last_became_peered":"2024-04-22T08:27:56.351763+0000","last_unstale":"2024-04-22T08:41:18.002600+0000","last_undegraded":"2024-04-22T08:41:18.002600+0000","last_fullsized":"2024-04-22T08:41:18.002600+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:18.002550+0000","last_clean_scrub_stamp":"2024-04-22T08:41:18.002550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:26.665118+0000","scrub_duration":0.0087780029999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'19750","reported_seq":20454,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:49.954241+0000","last_change":"2024-04-22T08:41:49.954462+0000","last_active":"2024-04-22T08:41:49.954241+0000","last_peered":"2024-04-22T08:41:49.954241+0000","last_clean":"2024-04-22T08:41:49.954241+0000","last_became_active":"2024-04-22T08:40:06.219585+0000","last_became_peered":"2024-04-22T08:40:06.219585+0000","last_unstale":"2024-04-22T08:41:49.954241+0000","last_undegraded":"2024-04-22T08:41:49.954241+0000","last_fullsized":"2024-04-22T08:41:49.954241+0000","mapping_epoch":54,"log_start":"37'9700","ondisk_log_start":"37'9700","created":12,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19750","last_scrub_stamp":"2024-04-22T08:41:49.954199+0000","last_deep_scrub":"41'19750","last_deep_scrub_stamp":"2024-04-22T08:41:49.954199+0000","last_clean_scrub_stamp":"2024-04-22T08:41:49.954199+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:16.351421+0000","scrub_duration":0.085102313999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15981,"num_write_kb":24440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":71,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:21.819490+0000","last_change":"2024-04-22T08:41:21.819490+0000","last_active":"2024-04-22T08:41:21.819490+0000","last_peered":"2024-04-22T08:41:21.819490+0000","last_clean":"2024-04-22T08:41:21.819490+0000","last_became_active":"2024-04-22T08:27:06.396762+0000","last_became_peered":"2024-04-22T08:27:06.396762+0000","last_unstale":"2024-04-22T08:41:21.819490+0000","last_undegraded":"2024-04-22T08:41:21.819490+0000","last_fullsized":"2024-04-22T08:41:21.819490+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:21.819114+0000","last_clean_scrub_stamp":"2024-04-22T08:41:21.819114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:52:32.713418+0000","scrub_duration":0.050105947999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'20038","reported_seq":20693,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:30.118394+0000","last_change":"2024-04-22T08:41:30.118394+0000","last_active":"2024-04-22T08:41:30.118394+0000","last_peered":"2024-04-22T08:41:30.118394+0000","last_clean":"2024-04-22T08:41:30.118394+0000","last_became_active":"2024-04-22T08:37:55.126170+0000","last_became_peered":"2024-04-22T08:37:55.126170+0000","last_unstale":"2024-04-22T08:41:30.118394+0000","last_undegraded":"2024-04-22T08:41:30.118394+0000","last_fullsized":"2024-04-22T08:41:30.118394+0000","mapping_epoch":44,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'20038","last_scrub_stamp":"2024-04-22T08:41:30.118102+0000","last_deep_scrub":"41'20038","last_deep_scrub_stamp":"2024-04-22T08:41:30.118102+0000","last_clean_scrub_stamp":"2024-04-22T08:41:30.118102+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:26:46.931464+0000","scrub_duration":0.066040161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":85,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:13.987251+0000","last_change":"2024-04-22T08:41:13.987493+0000","last_active":"2024-04-22T08:41:13.987251+0000","last_peered":"2024-04-22T08:41:13.987251+0000","last_clean":"2024-04-22T08:41:13.987251+0000","last_became_active":"2024-04-22T08:27:56.366682+0000","last_became_peered":"2024-04-22T08:27:56.366682+0000","last_unstale":"2024-04-22T08:41:13.987251+0000","last_undegraded":"2024-04-22T08:41:13.987251+0000","last_fullsized":"2024-04-22T08:41:13.987251+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:13.987199+0000","last_clean_scrub_stamp":"2024-04-22T08:41:13.987199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:57:16.122867+0000","scrub_duration":0.046424799000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:26.078271+0000","last_change":"2024-04-22T08:41:26.078439+0000","last_active":"2024-04-22T08:41:26.078271+0000","last_peered":"2024-04-22T08:41:26.078271+0000","last_clean":"2024-04-22T08:41:26.078271+0000","last_became_active":"2024-04-22T08:27:56.101474+0000","last_became_peered":"2024-04-22T08:27:56.101474+0000","last_unstale":"2024-04-22T08:41:26.078271+0000","last_undegraded":"2024-04-22T08:41:26.078271+0000","last_fullsized":"2024-04-22T08:41:26.078271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:41:26.078235+0000","last_clean_scrub_stamp":"2024-04-22T08:41:26.078235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:28.495349+0000","scrub_duration":0.064540178000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"59'20003","reported_seq":20767,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T08:41:32.176811+0000","last_change":"2024-04-22T08:41:32.177034+0000","last_active":"2024-04-22T08:41:32.176811+0000","last_peered":"2024-04-22T08:41:32.176811+0000","last_clean":"2024-04-22T08:41:32.176811+0000","last_became_active":"2024-04-22T08:38:57.186826+0000","last_became_peered":"2024-04-22T08:38:57.186826+0000","last_unstale":"2024-04-22T08:41:32.176811+0000","last_undegraded":"2024-04-22T08:41:32.176811+0000","last_fullsized":"2024-04-22T08:41:32.176811+0000","mapping_epoch":48,"log_start":"37'10000","ondisk_log_start":"37'10000","created":12,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'20003","last_scrub_stamp":"2024-04-22T08:41:32.176751+0000","last_deep_scrub":"59'20003","last_deep_scrub_stamp":"2024-04-22T08:41:32.176751+0000","last_clean_scrub_stamp":"2024-04-22T08:41:32.176751+0000","objects_scrubbed":1,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:36.014650+0000","scrub_duration":0.088016560999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173238,"num_write_kb":293649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":23015424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8876032,"data_stored":8869074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160617,"ondisk_log_size":160617,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":27,"seq":115964117178,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369724,"kb_used_data":9140,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":94002116,"statfs":{"total":96636764160,"available":96258166784,"internally_reserved":0,"allocated":9359360,"data_stored":9033784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369225423},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.58199999999999996,"15min":0.93000000000000005},"min":{"1min":0.26400000000000001,"5min":0.191,"15min":0.16300000000000001},"max":{"1min":0.58799999999999997,"5min":6.3860000000000001,"15min":13.635},"last":0.499},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.69199999999999995,"15min":0.81899999999999995},"min":{"1min":0.376,"5min":0.25800000000000001,"15min":0.18099999999999999},"max":{"1min":0.70099999999999996,"5min":6.431,"15min":13.721},"last":0.69599999999999995}]},{"osd":1,"last update":"Mon Apr 22 08:41:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47199999999999998,"15min":1.0089999999999999},"min":{"1min":0.254,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":1.419,"15min":15.6},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.76000000000000001,"15min":1.0289999999999999},"min":{"1min":0.27900000000000003,"5min":0.23400000000000001,"15min":0.20499999999999999},"max":{"1min":0.72799999999999998,"5min":6.6239999999999997,"15min":18.32},"last":0.63400000000000001}]}]},{"osd":1,"up_from":33,"seq":141733920939,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523500,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":523003,"kb_avail":93848340,"statfs":{"total":96636764160,"available":96100700160,"internally_reserved":0,"allocated":495616,"data_stored":166561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":535555788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56699999999999995,"15min":0.755},"min":{"1min":0.42299999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.84399999999999997,"5min":6.0629999999999997,"15min":9.0660000000000007},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.621,"15min":0.85599999999999998},"min":{"1min":0.57599999999999996,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":0.95799999999999996,"5min":4.2439999999999998,"15min":6.8860000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Apr 22 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.49199999999999999,"15min":1.099},"min":{"1min":0.44900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.78100000000000003,"5min":3.5950000000000002,"15min":15.989000000000001},"last":0.442},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":1.0589999999999999},"min":{"1min":0.442,"5min":0.23400000000000001,"15min":0.20699999999999999},"max":{"1min":0.76300000000000001,"5min":3.625,"15min":20.149000000000001},"last":0.54600000000000004}]}]},{"osd":0,"up_from":21,"seq":90194313407,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455332,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916508,"statfs":{"total":96636764160,"available":96170504192,"internally_reserved":0,"allocated":487424,"data_stored":164831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":465759955},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:41:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.68500000000000005,"15min":0.99299999999999999},"min":{"1min":0.192,"5min":0.17899999999999999,"15min":0.16},"max":{"1min":0.70899999999999996,"5min":10.435,"15min":11.368},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.91900000000000004},"min":{"1min":0.216,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.751,"5min":6.1740000000000004,"15min":19.658000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Apr 22 08:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.75800000000000001,"15min":0.93200000000000005},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.18099999999999999},"max":{"1min":0.78300000000000003,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.0649999999999999,"15min":1.0449999999999999},"min":{"1min":0.39500000000000002,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.78500000000000003,"5min":11.141999999999999,"15min":12.131},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8867840,"data_stored":8867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:42:12.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T08:42:12.879 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T08:42:12.879 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T08:42:12.879 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T08:42:12.879 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:18.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-22T08:42:18.981 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T08:42:18.981 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T08:42:18.981 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:24.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-22T08:42:24.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-22T08:42:25.083 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T08:42:25.083 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T08:42:25.083 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:29.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-22T08:42:29.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-22T08:42:31.185 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T08:42:31.185 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T08:42:31.185 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T08:42:31.186 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:31.215 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T08:42:31.216 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T08:42:31.216 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:42:31.216 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:31.276 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:42:31.276 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T08:42:31.276 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:31.339 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T08:42:31.339 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:42:31.339 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:42:31.398 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:42:31.399 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T08:42:31.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-04-22T08:42:31.515 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:42:31.516 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T08:42:31.803 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:42:31.803 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T08:42:32.051 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:42:32.052 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T08:42:32.314 INFO:tasks.ceph:Archiving mon data... 2024-04-22T08:42:32.316 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/data/mon.a.tgz 2024-04-22T08:42:32.316 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-22T08:42:32.340 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.RAIXki6Hyi 2024-04-22T08:42:32.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RAIXki6Hyi 2024-04-22T08:42:32.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.RAIXki6Hyi 2024-04-22T08:42:32.574 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.RAIXki6Hyi is 291KB 2024-04-22T08:42:32.632 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.RAIXki6Hyi 2024-04-22T08:42:32.644 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/data/mon.b.tgz 2024-04-22T08:42:32.644 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-22T08:42:32.691 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.aQJNgs6Seo 2024-04-22T08:42:32.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.aQJNgs6Seo 2024-04-22T08:42:32.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.aQJNgs6Seo 2024-04-22T08:42:32.875 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.aQJNgs6Seo is 291KB 2024-04-22T08:42:32.933 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.aQJNgs6Seo 2024-04-22T08:42:32.938 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/data/mon.c.tgz 2024-04-22T08:42:32.938 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-22T08:42:32.983 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.Ip5G7kXCmp 2024-04-22T08:42:32.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Ip5G7kXCmp 2024-04-22T08:42:33.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.Ip5G7kXCmp 2024-04-22T08:42:33.160 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.Ip5G7kXCmp is 287KB 2024-04-22T08:42:33.221 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.Ip5G7kXCmp 2024-04-22T08:42:33.226 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T08:42:33.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T08:42:33.300 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-22T08:42:33.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-22T08:42:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-22T08:42:33.369 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T08:42:33.371 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/remote/smithi049/crash 2024-04-22T08:42:33.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T08:42:33.420 INFO:tasks.ceph:Compressing logs... 2024-04-22T08:42:33.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:42:34.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-22T08:49:18.934 INFO:tasks.ceph:Archiving logs... 2024-04-22T08:49:18.935 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/remote/smithi049/log 2024-04-22T08:49:18.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T08:50:07.007 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T08:50:07.017 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T08:50:07.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T08:50:07.069 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T08:50:07.140 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T08:50:07.140 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout: remote refid st t when poll reach delay offset jitter 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================== 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout:*hv01.front.sepi 44.4.53.2 2 u 58 64 377 0.072 -8.489 0.496 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout:+hv02.front.sepi 63.231.80.2 3 u 59 64 377 0.078 -7.274 0.494 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.091 -7.769 0.622 2024-04-22T08:50:07.200 INFO:teuthology.orchestra.run.smithi049.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-22T08:50:07.202 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T08:50:07.241 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T08:50:07.242 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T08:50:07.251 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T08:50:07.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T08:50:07.267 INFO:teuthology.task.internal:Duration was 1996.590569 seconds 2024-04-22T08:50:07.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T08:50:07.276 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T08:50:07.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T08:50:07.464 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T08:50:07.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T08:50:07.465 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T08:50:07.476 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T08:50:07.477 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:50:07.558 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T08:50:07.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T08:50:07.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T08:50:07.733 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T08:50:07.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T08:50:07.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T08:50:07.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:50:07.832 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-04-22T08:50:07.852 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:50:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:50:07.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T08:50:07.909 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T08:50:07.910 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667875/remote/smithi049 2024-04-22T08:50:07.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T08:50:07.964 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T08:50:07.964 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T08:50:08.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T08:50:08.019 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T08:50:08.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T08:50:08.028 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T08:50:08.028 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T08:50:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: 658468 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 08:50 /home/ubuntu/cephtest 2024-04-22T08:50:08.058 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T08:50:08.068 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T08:50:08.243 INFO:teuthology.run:Summary data: description: rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} duration: 1996.5905690193176 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-22T08:50:08.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T08:50:08.332 INFO:teuthology.run:pass